如何在大型c ++ Linux应用程序中检查内存泄漏？

I am currently working on a large scale application project (written in c++) which started from scratch some time ago, and we reached the point when it is mandatory to make a roundup of checks for memory leaks.

我目前正在开发一个大规模的应用程序项目(用c ++编写),它从一开始就是从头开始的,我们已经到了强制要求对内存泄漏进行检查的时候了。

The application runs on an Ubuntu Linux, it has a lot of multimedia content, and uses OpenGl, SDL and ffmpeg for various purposes including 3D graph rendering, windows, audio and movie playback. You could think of it as a videogame, although it is not, but the duties of the application could be simplified by considering it a video game.

该应用程序在Ubuntu Linux上运行,它具有大量的多媒体内容,并且使用OpenGl,SDL和ffmpeg用于各种目的,包括3D图形渲染,窗口,音频和电影回放。您可以将其视为视频游戏,虽然它不是,但通过将视频游戏视为视频游戏,可以简化应用程序的职责。

I am currently a little bit clueless in determining whether we still have memory leaks or not. In the past we had already identified some, and removed them. These days though, the application is nearly complete, and the tests we ran are giving me results which I cant exactly figure out.

在确定我们是否还有内存泄漏时,我目前有点无能为力。在过去,我们已经确定了一些,并删除了它们。这些天,应用程序已接近完成,我们运行的测试给了我一些我无法弄清楚的结果。

First thing I did was to try to run the application through Valgrind... unfortunately then application crashes when running in a valgrind environment. The crash in "non-deterministic" since it crashes in various different places. So I gave up with Valgrind to easily identify the source of potential leaks, and ended up using two Linux commands: free and top.

我做的第一件事就是尝试通过Valgrind运行应用程序......不幸的是,在valgrind环境中运行时应用程序崩溃了。 “非确定性”崩溃,因为它在各个不同的地方崩溃。所以我放弃了Valgrind来轻松识别潜在泄漏的来源,并最终使用了两个Linux命令:free和top。

free is being used for probing system memory usage while the application is running

在应用程序运行时,free用于探测系统内存使用情况

top is being used with the '-p' option, to probe the application process memory usage while running.

top与'-p'选项一起使用,以在运行时探测应用程序进程内存使用情况。

Output form top and free is being dumped into files for post-processing. I made up two graphs with the data which are linked at the bottom of the question.

输出表单top和free将被转储到文件中以进行后处理。我用问题底部链接的数据组成了两个图表。

The test case is very simple: data about memory is being probed once the application has already been launched and it is waiting for commands. Then I start a sequence of commands which repeatedly does always the same thing. The application is expected to load a whole lot multimedia data into RAM, and then download it.

测试用例非常简单:一旦应用程序启动并等待命令,就会探测有关内存的数据。然后我开始一系列命令,这些命令反复执行相同的操作。该应用程序有望将大量多媒体数据加载到RAM中,然后下载。

Unfortunately the graph is not showing me what I was expecting. Memory usage grows through 3 different steps and then stops. Memory is apparently never released, which hinted me that there was a HUGE memory leak. that would be perfectly fine, since it would mean that very likely we were not freeing up memory eaten up by media stuff.

不幸的是,图表没有显示我的期望。内存使用量通过3个不同的步骤增长,然后停止。内存显然从未发布,这暗示我有一个巨大的内存泄漏。这将是非常好的,因为这意味着我们很可能没有释放被媒体吃掉的记忆。

But after the first three steps... memory usage is stable... there arent any more huge steps... just slight up and down which correspond to the expected data loading and unloading. The unexpected here is that the data which is supposed to be loaded/unloaded makes up for hundredths of megabytes of RAM, instead the up and downs make of for just a handful of megabytes (lets say 8-10 MB).

但是在前三个步骤之后...内存使用情况稳定...没有更大的步骤......只是轻微的上下,这对应于预期的数据加载和卸载。这里出乎意料的是,应该加载/卸载的数据占据了百万兆字节的RAM,而不是只有少数兆字节的数据(比如8-10 MB)。

I am currently pretty clueless in interpreting these data.

我目前对解释这些数据毫无头绪。

Anyone has some hints or suggestions? What am I missing? Is the method I am using for checking the presence of macroscopic memory leaks completely wrong? DO you know any other (preferably free) tool other than Valgrind for checking memory leaks?

任何人都有一些提示或建议吗?我错过了什么?我用来检查宏观内存泄漏是否完全错误的方法?你知道Valgrind以外的任何其他(最好是免费的)工具来检查内存泄漏吗?

System Memory Usage Graph

系统内存使用情况图

Process Memory Usage Graph

进程内存使用图

7 个解决方案

#1

First of all...

首先...

and we reached the point when it is mandatory to make a roundup of checks for memory leaks.

当我们必须对内存泄漏进行检查时,我们达到了这一点。

This, actually, is a problem of methodology. Correctness should be the primary goal of any software piece and not an afterthought.

实际上,这是方法论的问题。正确性应该是任何软件的主要目标,而不是事后的想法。

I will suppose though that you now realize this and how much easier it would have been to identify the problems had you been running an instrumented unit test at each commit.

我想虽然您现在已经意识到这一点,并且如果您在每次提交时都运行了一个检测单元测试,那么识别问题会更容易。

So, what to do now ?

那么,现在该怎么办?

Runtime detection:
- Try to make Valgrind work, you probably have some environmental issues
- Try ASan, ThreadSan and MemSan; they are not trivial to setup under Linux but oh so impressive!
- Try instrumented builds: tcmalloc includes a heap-checker for example
- ...

运行时检测:尝试让Valgrind工作,你可能有一些环境问题尝试ASan,ThreadSan和MemSan;它们在Linux下设置并不容易,但是令人印象深刻啊!尝试检测构建:tcmalloc包括一个堆检查器,例如...

Compile time detection:

编译时间检测:
- Turn on the warnings (preferably with -Werror) (not specific to your issue)
- Use static analysis, such as Clang's, it may spot unpaired allocation routines
- ...
Human detection:
- Code reviews: make sure all resources are allocated within RAII classes
- ...

人工检测:代码审查:确保所有资源都在RAII类中分配......

Note: using only RAII classes helps removing memory leaks, but does not help with dangling references. Thankfully, detecting dangling references is what ASan does.

注意:仅使用RAII类有助于消除内存泄漏,但对悬空引用没有帮助。值得庆幸的是,检测悬空参考是ASan所做的。

And once you have patched all the issues, make sure that this becomes part of the process. Changes should be reviewed and tested, always, so that rotten eggs are culled immediately rather than left to stink up the code base.

一旦您修复了所有问题,请确保这成为流程的一部分。应始终对变更进行审查和测试,以便立即剔除腐烂的鸡蛋,而不是让代码库变臭。

#2

Instead of giving up on Valgrind, you should instead work with them and try to

而不是放弃Valgrind,你应该与他们合作并尝试

get rid of the bugs you encountered in Valgrind

摆脱你在Valgrind遇到的错误

get your app thoroughly tested and debugged with the updated Valgrind.

使用更新的Valgrind对您的应用进行全面测试和调试。

Saying you gave up on Valgrind which is the solution to your problem isn't helping really...

说你放弃了Valgrind,这是你的问题的解决方案并没有帮助真正......

Valgrind is the tool we all use to check for memory-leaks and threading issues under linux.

Valgrind是我们用来检查linux下的内存泄漏和线程问题的工具。

In the end, it's definitely better to invest time in figuring out "why Valgrind doesn't work with my app" rather than looking for alternate solutions. Valgrind is a proved and tested tool, but not perfect. And it beats the alternative methods by a long, long shot.

最后,花时间搞清楚“为什么Valgrind不能与我的应用程序一起工作”而不是寻找替代解决方案当然更好。 Valgrind是经过验证和测试的工具,但并不完美。它通过长时间的长镜头击败了替代方法。

The Valgrind page says it's better to submit bugs to Bugzilla, but it's actually better to ask around on https://lists.sourceforge.net/lists/listinfo/valgrind-users if anyone saw such issues before and what to do in such a situation. Worst-case scenario - they'll tell you to file a bug to bugzilla or file it themselves.

Valgrind页面说最好向Bugzilla提交bug,但实际上最好在https://lists.sourceforge.net/lists/listinfo/valgrind-users上询问是否有人之前看过这样的问题以及在这样的问题上做什么情况。最坏情况 - 他们会告诉你向bugzilla提交bug或自行提交。

#3

You probably want to look at valgrind.

你可能想看看valgrind。

And you just may want to start with really simple examples to get a feel for what valgrind reports which can be somewhat verbose. Consider this simplified example where valgrind exactly what and how much is missing:

而你可能想要从非常简单的例子开始,以了解valgrind报告哪些有点冗长。考虑这个简化的例子,其中valgrind确切地缺少了什么和多少:

edd@max:/tmp$ cat valgrindex.cpp 

#include <cstdlib>

int main() {
  double *a = new double[100];
  exit(0);
}
edd@max:/tmp$ g++ -o valgrindex valgrindex.cpp 
edd@max:/tmp$ valgrind ./valgrindex
==15910== Memcheck, a memory error detector
==15910== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==15910== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==15910== Command: ./valgrindex
==15910== 
==15910== 
==15910== HEAP SUMMARY:
==15910==     in use at exit: 800 bytes in 1 blocks
==15910==   total heap usage: 1 allocs, 0 frees, 800 bytes allocated
==15910== 
==15910== LEAK SUMMARY:
==15910==    definitely lost: 0 bytes in 0 blocks
==15910==    indirectly lost: 0 bytes in 0 blocks
==15910==      possibly lost: 0 bytes in 0 blocks
==15910==    still reachable: 800 bytes in 1 blocks
==15910==         suppressed: 0 bytes in 0 blocks
==15910== Rerun with --leak-check=full to see details of leaked memory
==15910== 
==15910== For counts of detected and suppressed errors, rerun with: -v
==15910== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
edd@max:/tmp$

#4

Results from free and top will not be helpful to you. I regret that you put effort into constructing graphs for their results. I've given a good explanation why they are unhelpful in a similar topic here: Memory stability of a C++ application in Linux.

free和top的结果对你没有帮助。我很遗憾你努力为他们的结果构建图表。我已经给出了一个很好的解释,为什么它们在这里的类似主题中没有帮助:Linux中C ++应用程序的内存稳定性。

I will also concur with other answers here that you should probably prioritize troubleshooting the crash you're encountering in Valgrind. Valgrind is considered very stable at this point, and I have personally run rather complex multi-threaded multimedia SDL/OpenGL/etc. apps through it without issue. It is much more likely that Valgrind's running environment is exposing likely instabilities in your application. The crash sounds like a thread race condition crash, though it may also be heap/memory corruption.

我也同意这里的其他答案,你应该优先考虑解决你在Valgrind遇到的崩溃问题。 Valgrind在这一点上被认为非常稳定,我亲自运行了相当复杂的多线程多媒体SDL / OpenGL等。应用程序通过它没有问题。 Valgrind的运行环境更有可能暴露应用程序中可能存在的不稳定性。崩溃听起来像线程竞争条件崩溃,虽然它也可能是堆/内存损坏。

What you may want to ask for, then, is advice on how to debug an app that's crashing from within Valgrind's running environment (which is something I don't know the answer to).

那么,您可能想要问的是如何调试从Valgrind的运行环境中崩溃的应用程序的建议(这是我不知道的答案)。

#5

The trouble with free and top is they can show you a problem, but they give little help in fixing the problem. Of the 100's or 1000's of lines of code that allocate memory, which ones are leaking? This is where valgrind helps.

免费和*的问题是它们可以向您显示问题,但它们在解决问题时几乎没有帮助。在分配内存的100或1000行代码中,哪些代码正在泄漏?这是valgrind帮助的地方。

If this is for a company with a budget for tools, you might look at purify or other commercial tools.

如果这是针对具有工具预算的公司,您可以查看净化或其他商业工具。

Just for completeness I will mention the Boehm conservative garbage collecting memory allocator (which works for C and C++ code). You can turn off GC and use GC_Free() and it becomes a leak detection tool. Or you can leave GC enabled to automatically free memory when no longer used.

为了完整起见,我将提到Boehm保守垃圾收集内存分配器(适用于C和C ++代码)。您可以关闭GC并使用GC_Free(),它将成为泄漏检测工具。或者,您可以启用GC以在不再使用时自动释放内存。

#6

It all depends on which allocator you're using. The libc allocators (malloc, calloc, realloc) and the C++ allocators (new, delete) are probably using an optimization trick involving not releasing memory back to the OS. You see, if you ask malloc for some memory, use it, and then free it, it isn't necessarily being released back to the OS. Rather, if I ask malloc for memory, it is (most lily) getting much more then is necessary (because of page boundaries). That way, next time you need more memory, malloc just has it sitting around. Same thing with free. The memory is probably just added to mallocs memory pool, from which your later allocations are drawing.

这完全取决于您使用的分配器。 libc分配器(malloc,calloc,realloc)和C ++分配器(new,delete)可能正在使用一种优化技巧,包括不将内存释放回操作系统。你看,如果你向malloc请求一些内存,使用它,然后释放它,它不一定被释放回操作系统。相反,如果我要求malloc获取内存,那么(大多数百合)会得到更多,然后是必要的(因为页面边界)。这样,下次你需要更多内存时,malloc只是坐在那里。同样的事情是免费的。内存可能只是添加到mallocs内存池中,以后的分配正在从中进行绘制。

So, your applications first few mallocs put memory very high, but then the pool is big enough to accommodate future allocations.

所以,你的应用程序最初几个mallocs将内存放得很高,但随后池的大小足以容纳未来的分配。

#7

In addition of using valgrind you could also consider using Boehm's conservative GC; you probably want to compile it and configure it as a memory leak detector.

除了使用valgrind,您还可以考虑使用Boehm的保守GC;您可能想要编译它并将其配置为内存泄漏检测器。

And you might even dare using Boehm's GC as your primary memory allocator.

你甚至敢用Boehm的GC作为你的主要内存分配器。

BTW, looking into /proc/1234/maps could help you (where 1234 is the pid of your process).

顺便说一句,查看/ proc / 1234 / maps可以帮助你(1234是你的过程的pid)。

#1