为什么特定的UDP消息总是低于特定的缓冲区大小？

3 different messages are being sent to the same port at different rates:

3个不同的消息以不同的速率发送到同一个端口：

Message  size (bytes)  Sent everytransmit speed
High           232                 10 ms          100Hz                  
Medium     148                 20ms           50Hz                    
Low            20                   60 ms          16.6Hz                 

消息大小（字节）每次传输速度发送高232 10 ms 100Hz中148 148 20ms低20 60 ms 16.6Hz

I can only process one message every ~ 6 ms.
Single threaded. Blocking read.

我每隔约6毫秒只能处理一条消息。单线程。阻止阅读。

---

A strange situation is occurring, and I don't have an explanation for it.
When I set my receive buffer to 4,799 bytes, all of my low speed messages get dropped.
I see maybe one or two get processed, and then nothing.

一种奇怪的情况正在发生，我没有解释。当我将接收缓冲区设置为4,799字节时，我的所有低速消息都会被丢弃。我看到可能有一两个被处理，然后什么都没有。

When I set my receive buffer to 4,800(or higher!), it appears as though all of the low speed messages start getting processed. I see about 16/17 a second.

当我将接收缓冲区设置为4,800（或更高！）时，似乎所有低速消息都开始被处理。我看到大约每秒16/17。

---

This has been observed consistently. The application sending the packets is always started before the receiving application. The receiving application always has a long delay after the sockets are created, and before it begins processing. So the buffer is always full when the processing starts, and it is not the same starting buffer each time a test occurs. This is because the socket is created after the sender is already sending out messages, so the receiver might start listening in the middle of a send cycle.

这一点一直被观察到。发送数据包的应用程序始终在接收应用程序之前启动。创建套接字后，在开始处理之前，接收应用程序总是有很长的延迟。因此，处理开始时缓冲区始终为满，并且每次测试发生时它都不是相同的起始缓冲区。这是因为套接字是在发送方已经发送消息之后创建的，因此接收方可能会在发送周期的中间开始监听。

Why does increasing the received buffer size a single byte, cause a huge change in low speed message processing?

为什么增加单个字节的接收缓冲区大小会导致低速消息处理发生巨大变化？

I built a table to better visualize the expected processing:

我构建了一个表来更好地可视化预期的处理：

As some of these messages get processed, more messages presumably get put on the queue instead of being dropped.

由于其中一些消息得到处理，因此可能会将更多消息放入队列而不是被丢弃。

Nonetheless, I would expect a 4,799 byte buffer to behave the same way as 4,800 bytes.

尽管如此，我希望4,799字节缓冲区的行为与4,800字节相同。

However that is not what I have observed.

然而，这不是我所观察到的。

---

I think the issue is related to the fact that low speed messages are sent at the same time as the other two messages. It is always received after the high/medium speed messages. (This has been confirmed over wireshark).

我认为这个问题与低速消息与其他两条消息同时发送的事实有关。它总是在高/中速信息之后接收。 （已通过wireshark确认）。

For example, assuming the buffer was empty to begin with, it is clear that the low speed message would need queued longer than the other messages.
*1 message every 6ms is about 5 messages every 30ms.

例如，假设缓冲区开始时为空，很明显低速消息需要比其他消息排队更长。 *每6ms 1条消息大约每30ms发送5条消息。

This still doesn't explain the buffer size.

这仍然不能解释缓冲区大小。

We are running VxWorks, and using their sockLib, which is an implementation of Berkeley sockets. Here is a snippet of what our socket creation looks like:
SOCKET_BUFFER_SIZE is what I'm changing.

我们正在运行VxWorks，并使用他们的sockLib，它是Berkeley套接字的一个实现。这是我们的套接字创建的代码片段：SOCKET_BUFFER_SIZE是我正在改变的。

struct sockaddr_in tSocketAddress;                          // Socket address
int     nSocketAddressSize = sizeof(struct sockaddr_in);    // Size of socket address structure
int     nSocketOption = 0;

// Already created
if (*ptParameters->m_pnIDReference != 0)
    return FALSE;

// Create UDP socket
if ((*ptParameters->m_pnIDReference = socket(AF_INET, SOCK_DGRAM, 0)) == ERROR)
{
    // Error
    CreateSocketMessage(ptParameters, "CreateSocket: Socket create failed with error.");

    // Not successful
    return FALSE;
}

// Valid local address
if (ptParameters->m_szLocalIPAddress != SOCKET_ADDRESS_NONE_STRING && ptParameters->m_usLocalPort != 0)
{
    // Set up the local parameters/port
    bzero((char*)&tSocketAddress, nSocketAddressSize);
    tSocketAddress.sin_len = (u_char)nSocketAddressSize;
    tSocketAddress.sin_family = AF_INET;
    tSocketAddress.sin_port = htons(ptParameters->m_usLocalPort);

    // Check for any address
    if (strcmp(ptParameters->m_szLocalIPAddress, SOCKET_ADDRESS_ANY_STRING) == 0)
        tSocketAddress.sin_addr.s_addr = htonl(INADDR_ANY);
    else
    {
        // Convert IP address for binding
        if ((tSocketAddress.sin_addr.s_addr = inet_addr(ptParameters->m_szLocalIPAddress)) == ERROR)
        {
            // Error
            CreateSocketMessage(ptParameters, "Unknown IP address.");

            // Cleanup socket
            close(*ptParameters->m_pnIDReference);
            *ptParameters->m_pnIDReference = ERROR;

            // Not successful
            return FALSE;
        }
    }

    // Bind the socket to the local address
    if (bind(*ptParameters->m_pnIDReference, (struct sockaddr *)&tSocketAddress, nSocketAddressSize) == ERROR)
    {
        // Error
        CreateSocketMessage(ptParameters, "Socket bind failed.");

        // Cleanup socket
        close(*ptParameters->m_pnIDReference);
        *ptParameters->m_pnIDReference = ERROR;

        // Not successful
        return FALSE;
    }
}

// Receive socket
if (ptParameters->m_eType == SOCKTYPE_RECEIVE || ptParameters->m_eType == SOCKTYPE_RECEIVE_AND_TRANSMIT)
{
    // Set the receive buffer size
    nSocketOption = SOCKET_BUFFER_SIZE;
    if (setsockopt(*ptParameters->m_pnIDReference, SOL_SOCKET, SO_RCVBUF, (char *)&nSocketOption, sizeof(nSocketOption)) == ERROR)
    {
        // Error
        CreateSocketMessage(ptParameters, "Socket buffer size set failed.");

        // Cleanup socket
        close(*ptParameters->m_pnIDReference);
        *ptParameters->m_pnIDReference = ERROR;

        // Not successful
        return FALSE;
    }
}

and the socket receive that's being called in an infinite loop:
*The buffer size is definitely large enough

并且套接字接收在无限循环中调用：*缓冲区大小肯定足够大

int SocketReceive(int nSocketIndex, char *pBuffer, int nBufferLength)
{
    int nBytesReceived = 0;
    char szError[256];

    // Invalid index or socket
    if (nSocketIndex < 0 || nSocketIndex >= SOCKET_COUNT || g_pnSocketIDs[nSocketIndex] == 0)
    {
        sprintf(szError,"SocketReceive: Invalid socket (%d) or ID (%d)", nSocketIndex, g_pnSocketIDs[nSocketIndex]);
        perror(szError);
        return -1;
    }

    // Invalid buffer length
    if (nBufferLength == 0)
    {
        perror("SocketReceive: zero buffer length");
        return 0;
    }

    // Send data
    nBytesReceived = recv(g_pnSocketIDs[nSocketIndex], pBuffer, nBufferLength, 0);

    // Error in receiving
    if (nBytesReceived == ERROR)
    {
        // Create error string
        sprintf(szError, "SocketReceive: Data Receive Failure: <%d> ", errno);

        // Set error message
        perror(szError);

        // Return error
        return ERROR;
    }

    // Bytes received
    return nBytesReceived;
}

Any clues on why increasing the buffer size to 4,800 results in successful and consistent reading of low speed messages?

有关将缓冲区大小增加到4,800的原因的任何线索都可以成功一致地读取低速消息？

2 个解决方案

#1

The basic answer to the question of why a SO_RCVBUF size of 4799 results in lost low speed messages and a size of 4800 works fine is that with the mixture of the UDP packets coming in, the rate at which they are coming in, the rate at which you are processing incoming packets, and the sizing of the mbuff and cluster numbers in your vxWorks kernel allow for sufficient network stack throughput that the low speed messages are not being discarded with the larger size.

SO_RCVBUF大小为4799导致低速消息丢失和4800大小正常的问题的基本答案是UDP数据包的混合，它们进入的速率，速率您正在处理传入的数据包，并且您的vxWorks内核中的mbuff和簇编号的大小允许足够的网络堆栈吞吐量，以便不会丢弃较大的低速消息。

The SO_SNDBUF option description in the setsockopt() man page at URL http://www.vxdev.com/docs/vx55man/vxworks/ref/sockLib.html#setsockopt mentioned in a comment above has this to say about the size specified and the effect on mbuff usage:

上面评论中提到的URL http://www.vxdev.com/docs/vx55man/vxworks/ref/sockLib.html#setsockopt的setsockopt（）手册页中的SO_SNDBUF选项说明可以说明指定的大小和对mbuff使用的影响：

The effect of setting the maximum size of buffers (for both SO_SNDBUF and SO_RCVBUF, described below) is not actually to allocate the mbufs from the mbuf pool. Instead, the effect is to set the high-water mark in the protocol data structure, which is used later to limit the amount of mbuf allocation.

设置缓冲区的最大大小（对于SO_SNDBUF和SO_RCVBUF，如下所述）的效果实际上不是从mbuf池分配mbuf。相反，效果是在协议数据结构中设置高水位标记，稍后用于限制mbuf分配的数量。

UDP packets are discrete units. If you send 10 packets of size 232 that is not considered to be 2320 bytes of data in contiguous memory. Instead that is 10 memory buffers within the network stack because UDP is discrete packets while TCP is a continuous stream of bytes.

UDP数据包是离散单元。如果发送10个大小为232的数据包，则不会将其视为连续内存中的2320字节数据。相反，它是网络堆栈中的10个内存缓冲区，因为UDP是离散数据包，而TCP是连续的字节流。

See How do I tune the network buffering in VxWorks 5.4? in the DDS community web site which gives a discussion about the interdependence of the mixture of mbuff sizes and network clusters.

请参阅如何在VxWorks 5.4中调整网络缓冲？在DDS社区网站上，讨论了mbuff大小和网络集群混合的相互依赖性。

See How do I resolve a problem with VxWorks buffers? in the DDS community web site.

请参阅如何解决VxWorks缓冲区的问题？在DDS社区网站上。

See this PDF of a slide presentation, A New Tool to study Network Stack Exhaustion in VxWorks from 2004 which discusses using various tools such as mBufShow and inetStatShow to see what is happening in the network stack.

请参阅幻灯片演示文稿的PDF，这是一个研究2004年VxWorks中网络堆栈耗尽的新工具，它讨论了使用mBufShow和inetStatShow等各种工具来查看网络堆栈中发生的情况。

#2

Without detailed analysis of every network stack implementation along the path your UDP messages are being sent it is nearly impossible to state the resulting behaviour.

如果不对发送UDP消息的路径中的每个网络堆栈实现进行详细分析，则几乎不可能说明结果行为。

UDP implementations are allowed to drop any packet at their own discretion. Usually this happens when a stack comes to the conclusion that it would need to drop packets to be able to receive new ones. There is no formal requirement that the packets dropped are the oldest or the newest being received. It could also be that a certain size class is more affected due to internal memory management strategies.

UDP实现允许自行决定丢弃任何数据包。通常，当堆栈得出结论它需要丢弃数据包以便能够接收新数据包时，就会发生这种情况。没有正式要求丢弃的数据包是最旧的或最新的。也可能是由于内部存储器管理策略，特定大小的类更受影响。

From the IP stacks involved the most interesting one is the one on the receiving machine.

从涉及的IP堆栈中，最有趣的一个是接收机器上的那个。

For sure you will get better receive experience if you change the receive side to have a receive buffer that will take several seconds full of expected messages. I'd start with at least 10k.

如果您将接收方改为接收缓冲区需要几秒钟的预期消息，那么肯定会获得更好的接收体验。我从至少10k开始。

The observed "change" in behaviour when going from 4,799 to 4,800 may result from the later just allowing one of the small messages to be received before it needs to be dropped again, while the smaller size just causes it to be dropped slightly earlier. If the receiving application is quick enough to read the pending message you will receive small messages in the one case and no small messages in the other case.

从4,799到4,800时观察到的行为“变化”可能是由于后者只是允许在需要再次丢弃之前接收到一条小消息，而较小的大小只会导致它稍早丢弃。如果接收应用程序足够快以读取待处理消息，则在一种情况下将收到小消息，而在另一种情况下则不会收到小消息。