I have a large 2GB file with 1.5 million listings to process. I am running a console app that performs some string manipulation then uploads each listing to the database.
我有一个大的2GB文件,有150万个列表要处理。我正在运行一个控制台应用程序,它执行一些字符串操作,然后将每个列表上传到数据库。
-
I created a LINQ object and clear the object by assigning it to a new LinqObject() for each listing (loop).
我创建了一个LINQ对象,并通过为每个列表(循环)分配一个新的LinqObject()来清除该对象。
-
When the object is complete, I add it to a list.
对象完成后,我将其添加到列表中。
-
When the list reaches 100 objects, I submitAll on the entire list, clear the list, then repeat.
当列表达到100个对象时,我在整个列表上提交所有内容,清除列表,然后重复。
My memory usage continues to grow as the program runs. Is there anything I should be doing to keep memory usage down? I tried GC.collect. I think I want to use dispose..
程序运行时,我的内存使用量持续增长。我应该做些什么来减少内存使用量?我试过GC.collect。我想我想用处理..
Thanks in advance for looking.
提前谢谢你的期待。
3 个解决方案
#1
It's normal for the memory usage of a program to increase when it's working. You should not try to force the garbage collector to reduce the memory usage to try to save resources, this will most likely waste resources instead.
程序的内存使用量在正常工作时增加是正常的。您不应该尝试强制垃圾收集器减少内存使用以尝试节省资源,这很可能会浪费资源。
Contrary to one's first reaction, high memory usage is not a performance problem as long as there are any free memory left at all. Having a lot of unused memory doesn't increase the performance a bit. If you try to reduce the memory usage only to keep it down, you are just wasting CPU time doing cleanup that is not needed.
与一个人的第一反应相反,只要有任何空闲内存,高内存使用就不是性能问题。拥有大量未使用的内存并不会增加性能。如果你试图减少内存使用量只是为了降低内存使用率,那么你只是在浪费CPU时间进行不需要的清理工作。
If you are running out of free memory or if some other application needs it, the garbage collector will do the appropriate cleanup. In almost every situation the garbage collector will know much more about the current memory situatiuon than you can possibly anticipate when writing the code.
如果您的可用内存不足或某些其他应用程序需要它,垃圾收集器将进行适当的清理。在几乎所有情况下,垃圾收集器都会比编写代码时可能预期的更多地了解当前的内存情况。
If you are using objects that implement the IDisposable interface, you should call the Dispose method to free unmanaged resources, but all other objects are handled by the garbage collector. Managed objects normally don't leak memory at all.
如果您正在使用实现IDisposable接口的对象,则应调用Dispose方法以释放非托管资源,但所有其他对象都由垃圾回收器处理。托管对象通常根本不会泄漏内存。
#2
Do you need your memory usage to stay low? Absent an actual functional problem, high memory usage in and of itself is not an issue.
你需要你的内存使用率保持低吗?如果没有实际的功能问题,高内存使用本身就不是问题。
#3
How large is the memory usage growing? It may be that .NET is just "settling" effectively.
内存使用量有多大?可能是.NET只是有效地“解决”了。
It's not really clear exactly how you're doing this, but the general principle sounds okay. I suggest you take the database work out of the equation - just comment out whichever line would actually submit to the database. See how much memory that uses. Other than the StreamReader (or whatever) you shouldn't have anything else that needs disposing if you're not touching the database - just building batches of transformed objects and throwing them away.
目前还不清楚你是如何做到这一点的,但一般原则听起来还不错。我建议你把数据库工作排除在外 - 只需注释掉实际提交给数据库的那一行。查看使用了多少内存。除了StreamReader(或其他任何东西)之外,如果你没有触及数据库,你不应该有任何需要处理的东西 - 只需构建批量转换对象并扔掉它们。
#1
It's normal for the memory usage of a program to increase when it's working. You should not try to force the garbage collector to reduce the memory usage to try to save resources, this will most likely waste resources instead.
程序的内存使用量在正常工作时增加是正常的。您不应该尝试强制垃圾收集器减少内存使用以尝试节省资源,这很可能会浪费资源。
Contrary to one's first reaction, high memory usage is not a performance problem as long as there are any free memory left at all. Having a lot of unused memory doesn't increase the performance a bit. If you try to reduce the memory usage only to keep it down, you are just wasting CPU time doing cleanup that is not needed.
与一个人的第一反应相反,只要有任何空闲内存,高内存使用就不是性能问题。拥有大量未使用的内存并不会增加性能。如果你试图减少内存使用量只是为了降低内存使用率,那么你只是在浪费CPU时间进行不需要的清理工作。
If you are running out of free memory or if some other application needs it, the garbage collector will do the appropriate cleanup. In almost every situation the garbage collector will know much more about the current memory situatiuon than you can possibly anticipate when writing the code.
如果您的可用内存不足或某些其他应用程序需要它,垃圾收集器将进行适当的清理。在几乎所有情况下,垃圾收集器都会比编写代码时可能预期的更多地了解当前的内存情况。
If you are using objects that implement the IDisposable interface, you should call the Dispose method to free unmanaged resources, but all other objects are handled by the garbage collector. Managed objects normally don't leak memory at all.
如果您正在使用实现IDisposable接口的对象,则应调用Dispose方法以释放非托管资源,但所有其他对象都由垃圾回收器处理。托管对象通常根本不会泄漏内存。
#2
Do you need your memory usage to stay low? Absent an actual functional problem, high memory usage in and of itself is not an issue.
你需要你的内存使用率保持低吗?如果没有实际的功能问题,高内存使用本身就不是问题。
#3
How large is the memory usage growing? It may be that .NET is just "settling" effectively.
内存使用量有多大?可能是.NET只是有效地“解决”了。
It's not really clear exactly how you're doing this, but the general principle sounds okay. I suggest you take the database work out of the equation - just comment out whichever line would actually submit to the database. See how much memory that uses. Other than the StreamReader (or whatever) you shouldn't have anything else that needs disposing if you're not touching the database - just building batches of transformed objects and throwing them away.
目前还不清楚你是如何做到这一点的,但一般原则听起来还不错。我建议你把数据库工作排除在外 - 只需注释掉实际提交给数据库的那一行。查看使用了多少内存。除了StreamReader(或其他任何东西)之外,如果你没有触及数据库,你不应该有任何需要处理的东西 - 只需构建批量转换对象并扔掉它们。