有没有一种方法可以在不载入所有训练集的情况下训练一个神经网络?

I've been using a .NET framework to train a neural network, and my data set is really large, with a large number of inputs.

我一直在使用。net框架训练神经网络，我的数据集非常大，有大量的输入。

I've decided to switch to Encog due to some problems with the old framework I was using, and also because Encog seems much richer and offers additional features.

由于我使用的旧框架出现了一些问题，我决定转到Encog，而且Encog似乎更加丰富，提供了更多的特性。

My problem is that Encog loads all training data at once, into a 2D array, and this doesn't suit me since I have a very large data set. I've tried working around this, and also checked other questions (such as this one but couldn't find a good answer).

我的问题是，Encog将所有训练数据一次加载到一个2D数组中，这并不适合我，因为我有一个非常大的数据集。

I tried working with SQLNeuralDataSet and other implementations that support streaming operation but they all seem to eventually load the data into memory (using MemoryDataLoader) and this doesn't solve my problem.

我尝试使用SQLNeuralDataSet和其他支持流操作的实现，但它们似乎最终都将数据加载到内存中(使用MemoryDataLoader)，这并不能解决我的问题。

Is there a way I can load (and train) my network one item at a time? Or is this option not available in Encog?

是否有一种方法可以一次加载(和训练)我的网络一个项目?或者这个选项在伦敦奥组委没有?

Thanks

谢谢

EDIT

编辑

I ended up copying the source code and modifying it according to my needs. Basically, it all comes down to Process(IMLDataPair pair) in GradientWorker which, according to the documentation:

最后我复制了源代码并根据需要修改它。基本上，这一切都归结为在GradientWorker中处理(IMLDataPair对)，根据文档说明:

Process one training set element.

处理一个训练集元素。

1 个解决方案

#1

Looking at the source code for the data sources for Encog, there are a number of options. Look at Buffer/BufferedMLDataSet.cs, for example:

查看Encog的数据源的源代码，有很多选项。看看缓冲/ BufferedMLDataSet。cs,例如:

/// This class is not memory based, so very long files can be used, without
/// running out of memory. This dataset uses a Encog binary training file as a
/// buffer.