I've been trying to find information on this, but due to the immaturity of the Spring Integration framework I haven't had much luck.
我一直试图找到相关的信息,但由于Spring Integration框架的不成熟,我没有太多运气。
Here is my desired work flow:
这是我想要的工作流程:
-
New files are placed in an 'Incoming' directory
新文件放在“Incoming”目录中
-
Files are picked up using a file:inbound-channel-adapter
使用文件:inbound-channel-adapter拾取文件
-
The file content is streamed, N lines at a time, to a 'Stage 1' channel, which parses the line into an intermediary (shared) representation.
文件内容一次N行流到“阶段1”通道,该通道将该行解析为中间(共享)表示。
-
This parsed line is routed to multiple 'Stage 2' channels.
此解析的行路由到多个“Stage 2”通道。
-
Each 'Stage 2' channel does its own processing on the N available lines to convert them to a final representation. This channel must have a queue which ensures no Stage 2 channel is overwhelmed in the event that one channel processes significantly slower than the others.
每个“阶段2”通道在N个可用行上进行自己的处理,以将它们转换为最终表示。此通道必须具有一个队列,以确保在一个通道处理速度明显慢于其他通道的情况下,不会使第2阶段通道不堪重负。
-
The final representation of the N lines is written to a file. There will be as many output files as there were routing destinations in step 4.
N行的最终表示被写入文件。在步骤4中将存在与路由目的地一样多的输出文件。
**'N' above stands for any reasonable number of lines to read at a time, from [1, whatever I can fit into memory reasonably], but is guaranteed to always be less than the number of lines in the full file.*
**'N'代表一次读取任何合理数量的行,从[1,无论我能合理地放入内存中],但保证始终小于完整文件中的行数。*
How can I accomplish streaming (steps 3, 4, 5) in Spring Integration? It's fairly easy to do without streaming the files, but my files are large enough that I cannot read the entire file into memory.
如何在Spring Integration中完成流式传输(步骤3,4,5)?没有流式传输文件很容易,但我的文件足够大,我无法将整个文件读入内存。
As a side note, I have a working implementation of this work flow without Spring Integration, but since we're using Spring Integration in other places in our project, I'd like to try it here to see how it performs and how the resulting code compares for length and clarity.
作为旁注,我有一个没有Spring Integration的工作流程的工作实现,但由于我们在项目的其他地方使用Spring Integration,我想在这里尝试看看它是如何执行的以及结果如何代码比较长度和清晰度。
1 个解决方案
#1
1
This is a very interesting use case that I'm sorry I missed for such a long time. It's definitely worth creating an issue for. At the moment we have support in Spring Integration for picking up the files and sending references to them around. There is also some rudimentary support for converting the files to a byte[] or a String.
这是一个非常有趣的用例,我很抱歉我错过了这么长时间。绝对值得创造一个问题。目前,我们在Spring Integration中支持拾取文件并向其发送引用。将文件转换为byte []或String也有一些基本的支持。
The answer is that now you would do step 2 in custom java code, sending the chunks off to a Stage 2 channel. I would recommend against sending references to streams around as message payloads.
答案是,现在您将在自定义Java代码中执行第2步,将块发送到Stage 2通道。我建议不要将对流的引用作为消息有效负载发送。
#1
1
This is a very interesting use case that I'm sorry I missed for such a long time. It's definitely worth creating an issue for. At the moment we have support in Spring Integration for picking up the files and sending references to them around. There is also some rudimentary support for converting the files to a byte[] or a String.
这是一个非常有趣的用例,我很抱歉我错过了这么长时间。绝对值得创造一个问题。目前,我们在Spring Integration中支持拾取文件并向其发送引用。将文件转换为byte []或String也有一些基本的支持。
The answer is that now you would do step 2 in custom java code, sending the chunks off to a Stage 2 channel. I would recommend against sending references to streams around as message payloads.
答案是,现在您将在自定义Java代码中执行第2步,将块发送到Stage 2通道。我建议不要将对流的引用作为消息有效负载发送。