MPI_Isend和MPI_Irecv似乎导致死锁

I'm using non-blocking communication in MPI to send various messages between processes. However, I appear to be getting a deadlock. I have used PADB (see here) to look at the message queues and have got the following output:

我在MPI中使用非阻塞通信在进程之间发送各种消息。但是,我似乎陷入了僵局。我使用PADB(请参阅此处)查看消息队列并获得以下输出:

1:msg12: Operation 1 (pending_receive) status 0 (pending)
1:msg12: Rank local 4 global 4
1:msg12: Size desired 4
1:msg12: tag_wild 0
1:msg12: Tag desired 16
1:msg12: system_buffer 0
1:msg12: Buffer 0xcaad32c
1:msg12: 'Receive: 0xcac3c80'
1:msg12: 'Data: 4 * MPI_FLOAT'
--
1:msg32: Operation 0 (pending_send) status 2 (complete)
1:msg32: Rank local 4 global 4
1:msg32: Actual local 4 global 4
1:msg32: Size desired 4 actual 4
1:msg32: tag_wild 0
1:msg32: Tag desired 16 actual 16
1:msg32: system_buffer 0
1:msg32: Buffer 0xcaad32c
1:msg32: 'Send: 0xcab7c00'
1:msg32: 'Data transfer completed'
--
2:msg5: Operation 1 (pending_receive) status 0 (pending)
2:msg5: Rank local 1 global 1
2:msg5: Size desired 4
2:msg5: tag_wild 0
2:msg5: Tag desired 16
2:msg5: system_buffer 0
2:msg5: Buffer 0xabbc348
2:msg5: 'Receive: 0xabd1780'
2:msg5: 'Data: 4 * MPI_FLOAT'
--
2:msg25: Operation 0 (pending_send) status 2 (complete)
2:msg25: Rank local 1 global 1
2:msg25: Actual local 1 global 1
2:msg25: Size desired 4 actual 4
2:msg25: tag_wild 0
2:msg25: Tag desired 16 actual 16
2:msg25: system_buffer 0
2:msg25: Buffer 0xabbc348
2:msg25: 'Send: 0xabc5700'
2:msg25: 'Data transfer completed'

This seems to have showed that sends have completed, but all of the receives are pending (the above is just an small part of the log for a tag value of 16). However, how can this happen? Surely sends can't complete without the associated receive completing, as in MPI all sends and receives have to match. At least that's what I thought...

这似乎已经表明发送已经完成,但所有接收都是挂起的(上面只是标记值为16的日志的一小部分)。但是,这怎么可能发生呢?肯定发送不能完成没有相关的接收完成,因为在MPI中所有发送和接收都必须匹配。至少那是我的想法......

Can anyone provide any insights?

任何人都可以提供任何见解吗?

I can provide the code I'm using to do this, but surely Isend and Irecv should work regardless of what order they are all called in, assuming that MPI_Waitall is called right at the end.

我可以提供我正在使用的代码,但是Isend和Irecv肯定应该工作,无论它们被调用的顺序如何,假设最后调用MPI_Waitall。

Update: Code is available at this gist

更新:此要点提供代码

Update: I've made various modifications to the code, but it still isn't working quite properly. The new code is at the same gist, and the output I'm getting is at this gist. I have a number of questions/issues with this code:

更新:我对代码进行了各种修改,但它仍然无法正常工作。新代码在同一个要点上,我得到的输出就是这个要点。我对此代码有许多疑问/问题:

1. Why is the output from the final loop (printing all of the arrays) interspersed with the rest of the output when I have a MPI_Barrier() before it to make sure all of the work has been done before printing it out?

   为什么在我有一个MPI_Barrier()之前,最后一个循环的输出(打印所有数组)散布在输出的其余部分,以确保在打印之前完成所有工作?

2. It is possible/sensible to be sending from rank 0 to rank 0 - will that work ok? (assuming a correct matching receive is posted, of course).

   从0级发送到0级是可行/明智的 - 这样可行吗? (假设发布了正确的匹配接收,当然)。

3. I'm getting lots of very strange long numbers in the output, which I assume is some kinda of memory-overwriting problem, or sizes of variables problem. The interesting thing is that this must be resulting from the MPI communications, because I initialise new_array to a value of 9999.99 and the communication obviously causes it to be changed to these strange values. Any ideas why?

   我在输出中得到了很多非常奇怪的长数,我认为这是一些内存覆盖问题,或者变量大小问题。有趣的是,这必须是由MPI通信产生的,因为我将new_array初始化为值9999.99并且通信显然导致它被更改为这些奇怪的值。有什么想法吗?

Overall it seems that some of the transposition is occurring (bits of the matrix seem to be transposed...), but definitely not all of it - it's these strange numbers that are coming up that are worrying me the most!

总的来说,似乎有一些转置正在发生(矩阵的位似乎被转换......),但绝对不是全部 - 这些正在出现的奇怪数字让我最担心!

1 个解决方案

#1

When using MPI_Isend and MPI_Irecv you have to be sure to not modify the buffers before you wait for the request to complete, and you are definitely violating this. What if you had the recieves all go into a second matrix instead of doing it in place?

使用MPI_Isend和MPI_Irecv时,您必须确保在等待请求完成之前不修改缓冲区,并且您肯定违反了此规则。如果您将所有接收器全部放入第二个矩阵而不是将其放置到位,该怎么办?

Also, global_x2 * global_y2 is your tag, but I'm not sure that it will be unique for every send-recieve pair, which could be messing things up. What happens if you switch it to sending tag (global_y2 * global_columns) + global_x2 and recieving tag (global_x2 * global_columns) + global_y2.

另外,global_x2 * global_y2是你的标签,但我不确定它对于每个发送 - 接收对都是唯一的,这可能会搞砸了。如果将其切换为发送标记(global_y2 * global_columns)+ global_x2和接收标记(global_x2 * global_columns)+ global_y2会发生什么情况。

Edit: As for your question about output, I'm assuming you are testing this by running all your processes on the same machine and just looking at the standard output. When you do it this way, your output gets buffered oddly by the terminal, even though the printf code all executes before the barrier. There are two ways I get around this. You could either print to a separate file for each process, or you could send your output as messages to process 0 and let him do all the actual printing.

编辑:至于你关于输出的问题,我假设你是通过在同一台机器上运行所有进程并只看标准输出来测试它。当你这样做时,你的输出会被终端奇怪地缓冲,即使printf代码都在屏障之前执行。我有两种解决方法。您可以为每个进程打印到单独的文件,也可以将输出作为消息发送到进程0并让他进行所有实际打印。