My goal is to send/share data between multiple programs. These are the options I thought of:
我的目标是在多个程序之间发送/共享数据。这些是我想到的选项:
- I could use a file, but prefer to use my RAM because it's generally faster.
- I could use a socket, but that would require a lot of address information which is unnecessary for local stuff. And ports too.
- I could ask others about an efficient way to do this.
我可以使用一个文件,但更喜欢使用我的RAM,因为它通常更快。
我可以使用套接字,但这需要大量的地址信息,这对于本地的东西是不必要的。还有端口。
我可以问别人一个有效的方法来做到这一点。
I chose the last one.
我选择了最后一个。
So, what would be an efficient way to send data from one program to another? It might use a buffer, for example, and write bytes to it and wait for the reciever to mark the first byte as 'read' (basically anything else than the byte written), then write again, but where would I put the buffer and how would I make it accessible for both programs? Or perhaps something else might work too?
那么,将数据从一个程序发送到另一个程序的有效方法是什么?例如,它可能使用缓冲区并向其写入字节并等待接收方将第一个字节标记为“读取”(基本上除了写入的字节之外的任何内容),然后再次写入,但是我将放置缓冲区和我怎样才能让两个程序都可以访问它?或许其他东西也可能起作用?
I use linux.
我用linux。
6 个解决方案
#1
4
What about fifos and pipes? if you are on a linux environment, this is the way to allow 2 programs to share data.
怎么样的fifos和管道?如果你在Linux环境中,这是允许2个程序共享数据的方法。
#2
4
The fastest IPC for processes running on same host is a shared memory.
在同一主机上运行的进程的最快IPC是共享内存。
In short, several processes can access same memory segment.
简而言之,几个进程可以访问相同的内存段。
See this tutorial.
请参阅本教程。
#3
3
You may want to take a look at Boost.Interprocess
您可能想看一下Boost.Interprocess
Boost.Interprocess simplifies the use of common interprocess communication and synchronization mechanisms and offers a wide range of them:
Boost.Interprocess简化了通用进程间通信和同步机制的使用,并提供了大量的:
Shared memory.
Memory-mapped files.
Semaphores, mutexes, condition variables and upgradable mutex types to place them in shared memory and memory mapped files.
信号量,互斥量,条件变量和可升级的互斥锁类型,用于将它们放在共享内存和内存映射文件中。
Named versions of those synchronization objects, similar to UNIX/Windows sem_open/CreateSemaphore API.
这些同步对象的命名版本,类似于UNIX / Windows sem_open / CreateSemaphore API。
File locking.
Relative pointers.
Message queues.
#4
2
To answer your questions:
回答你的问题:
Using a file is probably not the best way, and files are usually not used for passing inner-process information. Remember the os has to open, read, write, close them. They are however used for locking (http://en.wikipedia.org/wiki/File_locking).
使用文件可能不是最好的方法,文件通常不用于传递内部进程信息。记住操作系统必须打开,读取,写入,关闭它们。然而,它们用于锁定(http://en.wikipedia.org/wiki/File_locking)。
The highest performance you get using pipestream (http://linux.die.net/man/3/popen), but in Linux it's hard to get right. You have to redirect the stdin, stdout, and stderr. This has to be done for each inner-process. So it will work well for two applications but go beyond that and it gets very hairy.
使用pipestream获得的最高性能(http://linux.die.net/man/3/popen),但在Linux中很难做到正确。您必须重定向stdin,stdout和stderr。必须为每个内部过程完成此操作。所以它适用于两个应用程序,但超出了它,它变得非常毛茸茸。
My favorite solution, use socketpairs (http://pubs.opengroup.org/onlinepubs/009604499/functions/socketpair.html). These are very robust and easy to setup. But if you use multiple applications you have to prepare some sort of pool where to access the applications.
我最喜欢的解决方案,使用socketpairs(http://pubs.opengroup.org/onlinepubs/009604499/functions/socketpair.html)。这些非常强大且易于设置。但是,如果您使用多个应用程序,则必须准备某种池以访问应用程序。
#5
2
On Linux, when using files, they are very often in cache, so you won't read the disk that often, and you could use a "RAM" filesystem like tmpfs (actually tmpfs use virtual memory, so RAM + swap, and practically the files are kept in RAM most of the time).
在Linux上,当使用文件时,它们经常在缓存中,所以你不会经常读取磁盘,你可以使用像tmpfs这样的“RAM”文件系统(实际上tmpfs使用虚拟内存,所以RAM +交换,实际上文件大多数时间都保存在RAM中。
The main issue remains synchronization.
主要问题仍然是同步。
Using sockets (which may be, if all processes are on the same machine, AF_UNIX sockets which are faster than TCP/IP ones) has the advantage of making our code easily portable to environments where you prefer to run several processes on several machines.
使用套接字(可能是,如果所有进程都在同一台机器上,AF_UNIX套接字比TCP / IP套接字快),这样可以使我们的代码可以轻松移植到您希望在多台机器上运行多个进程的环境中。
And you could also use an existing framework for parallel execution, like e.g. MPI, Corba, etc etc.
您也可以使用现有框架进行并行执行,例如MPI,Corba等
You should have a gross idea of the bandwidth and latency expected from your application. (it is not the same if you need to share dozens of megabytes every millisecond, or hundreds of kilobytes every tenths of seconds).
您应该大致了解应用程序所期望的带宽和延迟。 (如果你需要每毫秒分享数十兆字节,或者每十分之几秒分享数百千字节,那就不一样了)。
I would suggest learning more about serialization techniques, formats and libraries like XDR, ASN1, JSON, YAML, s11n, jsoncpp etc.
我建议学习更多关于序列化技术,格式和库,如XDR,ASN1,JSON,YAML,s11n,jsoncpp等。
And sending or sharing data is not the same. When you send (and recieve) data, you think in terms of message passing. When you share data you think in terms of a shared memory. Programming style is very different.
发送或共享数据不一样。当您发送(并接收)数据时,您会考虑传递消息。当您共享数据时,您会考虑共享内存。编程风格非常不同。
#6
0
Shared memory is the best for sharing the data between the processes. But it needs lots of synchronization and if more than 2 processes are sharing the data then synchronization is like a Cyclops. (Single eye - Single shared memory).
共享内存最适合在进程之间共享数据。但它需要大量的同步,如果超过2个进程共享数据,那么同步就像独眼巨人。 (单眼 - 单个共享内存)。
But if you make use of sockets (multicast sockets), then implementation will be little difficult, but scalability and maintainability is very easy. You no need to bother how many apps will be waiting for the data, you can just multicast and they will listen to the data and process. No need to wait for the semaphore (shared memory synchronization technique) to read the data.
但是如果你使用套接字(多播套接字),那么实现起来就不会那么困难,但可扩展性和可维护性非常容易。您无需费心等待有多少应用程序等待数据,您可以只进行多播,他们将监听数据和进程。无需等待信号量(共享内存同步技术)来读取数据。
So reading the data time can be reduced.
Shared memory - Wait for the semaphore, read the data and process the data.
Sockets - Receive the data, process the data.
因此可以减少读取数据的时间。共享内存 - 等待信号量,读取数据并处理数据。套接字 - 接收数据,处理数据。
Performance, scalability and maintainability will be added advantages with the sockets.
使用套接字将增加性能,可伸缩性和可维护性。
Regards,
SSuman185
#1
4
What about fifos and pipes? if you are on a linux environment, this is the way to allow 2 programs to share data.
怎么样的fifos和管道?如果你在Linux环境中,这是允许2个程序共享数据的方法。
#2
4
The fastest IPC for processes running on same host is a shared memory.
在同一主机上运行的进程的最快IPC是共享内存。
In short, several processes can access same memory segment.
简而言之,几个进程可以访问相同的内存段。
See this tutorial.
请参阅本教程。
#3
3
You may want to take a look at Boost.Interprocess
您可能想看一下Boost.Interprocess
Boost.Interprocess simplifies the use of common interprocess communication and synchronization mechanisms and offers a wide range of them:
Boost.Interprocess简化了通用进程间通信和同步机制的使用,并提供了大量的:
Shared memory.
Memory-mapped files.
Semaphores, mutexes, condition variables and upgradable mutex types to place them in shared memory and memory mapped files.
信号量,互斥量,条件变量和可升级的互斥锁类型,用于将它们放在共享内存和内存映射文件中。
Named versions of those synchronization objects, similar to UNIX/Windows sem_open/CreateSemaphore API.
这些同步对象的命名版本,类似于UNIX / Windows sem_open / CreateSemaphore API。
File locking.
Relative pointers.
Message queues.
#4
2
To answer your questions:
回答你的问题:
Using a file is probably not the best way, and files are usually not used for passing inner-process information. Remember the os has to open, read, write, close them. They are however used for locking (http://en.wikipedia.org/wiki/File_locking).
使用文件可能不是最好的方法,文件通常不用于传递内部进程信息。记住操作系统必须打开,读取,写入,关闭它们。然而,它们用于锁定(http://en.wikipedia.org/wiki/File_locking)。
The highest performance you get using pipestream (http://linux.die.net/man/3/popen), but in Linux it's hard to get right. You have to redirect the stdin, stdout, and stderr. This has to be done for each inner-process. So it will work well for two applications but go beyond that and it gets very hairy.
使用pipestream获得的最高性能(http://linux.die.net/man/3/popen),但在Linux中很难做到正确。您必须重定向stdin,stdout和stderr。必须为每个内部过程完成此操作。所以它适用于两个应用程序,但超出了它,它变得非常毛茸茸。
My favorite solution, use socketpairs (http://pubs.opengroup.org/onlinepubs/009604499/functions/socketpair.html). These are very robust and easy to setup. But if you use multiple applications you have to prepare some sort of pool where to access the applications.
我最喜欢的解决方案,使用socketpairs(http://pubs.opengroup.org/onlinepubs/009604499/functions/socketpair.html)。这些非常强大且易于设置。但是,如果您使用多个应用程序,则必须准备某种池以访问应用程序。
#5
2
On Linux, when using files, they are very often in cache, so you won't read the disk that often, and you could use a "RAM" filesystem like tmpfs (actually tmpfs use virtual memory, so RAM + swap, and practically the files are kept in RAM most of the time).
在Linux上,当使用文件时,它们经常在缓存中,所以你不会经常读取磁盘,你可以使用像tmpfs这样的“RAM”文件系统(实际上tmpfs使用虚拟内存,所以RAM +交换,实际上文件大多数时间都保存在RAM中。
The main issue remains synchronization.
主要问题仍然是同步。
Using sockets (which may be, if all processes are on the same machine, AF_UNIX sockets which are faster than TCP/IP ones) has the advantage of making our code easily portable to environments where you prefer to run several processes on several machines.
使用套接字(可能是,如果所有进程都在同一台机器上,AF_UNIX套接字比TCP / IP套接字快),这样可以使我们的代码可以轻松移植到您希望在多台机器上运行多个进程的环境中。
And you could also use an existing framework for parallel execution, like e.g. MPI, Corba, etc etc.
您也可以使用现有框架进行并行执行,例如MPI,Corba等
You should have a gross idea of the bandwidth and latency expected from your application. (it is not the same if you need to share dozens of megabytes every millisecond, or hundreds of kilobytes every tenths of seconds).
您应该大致了解应用程序所期望的带宽和延迟。 (如果你需要每毫秒分享数十兆字节,或者每十分之几秒分享数百千字节,那就不一样了)。
I would suggest learning more about serialization techniques, formats and libraries like XDR, ASN1, JSON, YAML, s11n, jsoncpp etc.
我建议学习更多关于序列化技术,格式和库,如XDR,ASN1,JSON,YAML,s11n,jsoncpp等。
And sending or sharing data is not the same. When you send (and recieve) data, you think in terms of message passing. When you share data you think in terms of a shared memory. Programming style is very different.
发送或共享数据不一样。当您发送(并接收)数据时,您会考虑传递消息。当您共享数据时,您会考虑共享内存。编程风格非常不同。
#6
0
Shared memory is the best for sharing the data between the processes. But it needs lots of synchronization and if more than 2 processes are sharing the data then synchronization is like a Cyclops. (Single eye - Single shared memory).
共享内存最适合在进程之间共享数据。但它需要大量的同步,如果超过2个进程共享数据,那么同步就像独眼巨人。 (单眼 - 单个共享内存)。
But if you make use of sockets (multicast sockets), then implementation will be little difficult, but scalability and maintainability is very easy. You no need to bother how many apps will be waiting for the data, you can just multicast and they will listen to the data and process. No need to wait for the semaphore (shared memory synchronization technique) to read the data.
但是如果你使用套接字(多播套接字),那么实现起来就不会那么困难,但可扩展性和可维护性非常容易。您无需费心等待有多少应用程序等待数据,您可以只进行多播,他们将监听数据和进程。无需等待信号量(共享内存同步技术)来读取数据。
So reading the data time can be reduced.
Shared memory - Wait for the semaphore, read the data and process the data.
Sockets - Receive the data, process the data.
因此可以减少读取数据的时间。共享内存 - 等待信号量,读取数据并处理数据。套接字 - 接收数据,处理数据。
Performance, scalability and maintainability will be added advantages with the sockets.
使用套接字将增加性能,可伸缩性和可维护性。
Regards,
SSuman185