close()没有正确地关闭套接字。

I have a multi-threaded server (thread pool) that is handling a large number of requests (up to 500/sec for one node), using 20 threads. There's a listener thread that accepts incoming connections and queues them for the handler threads to process. Once the response is ready, the threads then write out to the client and close the socket. All seemed to be fine until recently, a test client program started hanging randomly after reading the response. After a lot of digging, it seems that the close() from the server is not actually disconnecting the socket. I've added some debugging prints to the code with the file descriptor number and I get this type of output.

我有一个多线程服务器(线程池)，它使用20个线程处理大量请求(一个节点高达500/秒)。有一个侦听器线程接受传入的连接，并将它们用于处理程序线程。一旦响应就绪，线程就会写入到客户机并关闭套接字。一切似乎都很好，直到最近，一个测试客户端程序在读取响应后开始随机地挂起。经过大量挖掘之后，似乎服务器的close()实际上并没有断开连接。我已经在代码中添加了一些调试打印文件，并使用文件描述符编号，我得到了这种类型的输出。

Processing request for 21
Writing to 21
Closing 21

The return value of close() is 0, or there would be another debug statement printed. After this output with a client that hangs, lsof is showing an established connection.

close()的返回值为0，否则将会出现另一个调试语句。在此输出与挂起的客户端之后，lsof显示了已建立的连接。

SERVER 8160 root 21u IPv4 32754237 TCP localhost:9980->localhost:47530 (ESTABLISHED)

服务器8160根21u IPv4 32754237 TCP localhost:9980->localhost:47530(已建立)

CLIENT 17747 root 12u IPv4 32754228 TCP localhost:47530->localhost:9980 (ESTABLISHED)

客户端17747根12u IPv4 32754228 TCP localhost:47530->localhost:9980(已建立)

It's as if the server never sends the shutdown sequence to the client, and this state hangs until the client is killed, leaving the server side in a close wait state

这就好像服务器从来没有将关机序列发送给客户机，而这个状态一直挂着，直到客户端被杀死，服务器端处于关闭状态。

SERVER 8160 root 21u IPv4 32754237 TCP localhost:9980->localhost:47530 (CLOSE_WAIT)

服务器8160根21u IPv4 32754237 TCP localhost:9980->localhost:47530 (CLOSE_WAIT)

Also if the client has a timeout specified, it will timeout instead of hanging. I can also manually run

另外，如果客户机指定了超时，它将超时而不是挂起。我也可以手动运行。

call close(21)

in the server from gdb, and the client will then disconnect. This happens maybe once in 50,000 requests, but might not happen for extended periods.

在来自gdb的服务器中，客户端将断开连接。这可能在5万次请求中发生一次，但可能不会在较长时间内发生。

Linux version: 2.6.21.7-2.fc8xen Centos version: 5.4 (Final)

Linux版本:2.6.21.7-2。fc8xen Centos版本:5.4 (Final)

socket actions are as follows

套接字操作如下。

SERVER:

服务器:

int client_socket; struct sockaddr_in client_addr; socklen_t client_len = sizeof(client_addr);

int client_socket;结构指向sockaddr_in client_addr;socklen_t client_len = sizeof(client_addr);

while(true) {
  client_socket = accept(incoming_socket, (struct sockaddr *)&client_addr, &client_len);
  if (client_socket == -1)
    continue;
  /*  insert into queue here for threads to process  */
}

Then the thread picks up the socket and builds the response.

然后线程获取套接字并构建响应。

/*  get client_socket from queue  */

/*  processing request here  */

/*  now set to blocking for write; was previously set to non-blocking for reading  */
int flags = fcntl(client_socket, F_GETFL);
if (flags < 0)
  abort();
if (fcntl(client_socket, F_SETFL, flags|O_NONBLOCK) < 0)
  abort();

server_write(client_socket, response_buf, response_length);
server_close(client_socket);

server_write and server_close.

server_write server_close。

void server_write( int fd, char const *buf, ssize_t len ) {
    printf("Writing to %d\n", fd);
    while(len > 0) {
      ssize_t n = write(fd, buf, len);
      if(n <= 0)
        return;// I don't really care what error happened, we'll just drop the connection
      len -= n;
      buf += n;
    }
  }

void server_close( int fd ) {
    for(uint32_t i=0; i<10; i++) {
      int n = close(fd);
      if(!n) {//closed successfully                                                                                                                                   
        return;
      }
      usleep(100);
    }
    printf("Close failed for %d\n", fd);
  }

CLIENT:

客户:

Client side is using libcurl v 7.27.0

客户端使用libcurl v7.27.0。

CURL *curl = curl_easy_init();
CURLcode res;
curl_easy_setopt( curl, CURLOPT_URL, url);
curl_easy_setopt( curl, CURLOPT_WRITEFUNCTION, write_callback );
curl_easy_setopt( curl, CURLOPT_WRITEDATA, write_tag );

res = curl_easy_perform(curl);

Nothing fancy, just a basic curl connection. Client hangs in tranfer.c (in libcurl) because the socket is not perceived as being closed. It's waiting for more data from the server.

没什么奇特的，只是一个基本的旋度连接。客户端挂在转变。c(在libcurl中)，因为该套接字不被认为是关闭的。它正在等待来自服务器的更多数据。

Things I've tried so far:

到目前为止我尝试过的事情:

Shutdown before close

关闭关闭之前

shutdown(fd, SHUT_WR);                                                                                                                                            
char buf[64];                                                                                                                                                     
while(read(fd, buf, 64) > 0);                                                                                                                                         
/*  then close  */

Setting SO_LINGER to close forcibly in 1 second

设置SO_LINGER在1秒内强制关闭。

struct linger l;
l.l_onoff = 1;
l.l_linger = 1;
if (setsockopt(client_socket, SOL_SOCKET, SO_LINGER, &l, sizeof(l)) == -1)
  abort();

These have made no difference. Any ideas would be greatly appreciated.

这些都没有影响。如有任何意见，我们将不胜感激。

EDIT -- This ended up being a thread-safety issue inside a queue library causing the socket to be handled inappropriately by multiple threads.

编辑——这最终成为队列库中的线程安全问题，导致多个线程不恰当地处理套接字。

3 个解决方案

#1

Here is some code I've used on many Unix-like systems (e.g SunOS 4, SGI IRIX, HPUX 10.20, CentOS 5, Cygwin) to close a socket:

下面是我在许多类unix系统上使用的一些代码。gsunos 4, SGI IRIX, hpux10.20, CentOS 5, Cygwin)关闭一个插座:

int getSO_ERROR(int fd) {
   int err = 1;
   socklen_t len = sizeof err;
   if (-1 == getsockopt(fd, SOL_SOCKET, SO_ERROR, (char *)&err, &len))
      FatalError("getSO_ERROR");
   if (err)
      errno = err;              // set errno to the socket SO_ERROR
   return err;
}

void closeSocket(int fd) {      // *not* the Windows closesocket()
   if (fd >= 0) {
      getSO_ERROR(fd); // first clear any errors, which can cause close to fail
      if (shutdown(fd, SHUT_RDWR) < 0) // secondly, terminate the 'reliable' delivery
         if (errno != ENOTCONN && errno != EINVAL) // SGI causes EINVAL
            Perror("shutdown");
      if (close(fd) < 0) // finally call close()
         Perror("close");
   }
}

But the above does not guarantee that any buffered writes are sent.

但是上面的内容并不能保证发送任何缓冲写。

Graceful close: It took me about 10 years to figure out how to close a socket. But for another 10 years I just lazily called usleep(20000) for a slight delay to 'ensure' that the write buffer was flushed before the close. This obviously is not very clever, because:

优雅的结尾:我花了大约10年的时间才弄清楚如何关闭一个插座。但在接下来的10年里，我只是懒散地给usleep(20000)打了个电话，以“确保”写缓冲区在关闭前被刷新。这显然不是很聪明，因为:

The delay was too long most of the time.
拖延时间太长了。
The delay was too short some of the time--maybe!
耽误的时间太短了——也许吧!
A signal such SIGCHLD could occur to end usleep() (but I usually called usleep() twice to handle this case--a hack).
一个信号这样的SIGCHLD可能会发生在usleep()上(但我通常会两次调用usleep()来处理这个例子——一个hack)。
There was no indication whether this works. But this is perhaps not important if a) hard resets are perfectly ok, and/or b) you have control over both sides of the link.
没有迹象表明这是否有效。但是这可能并不重要，如果一个)硬的重置是完全可以的，并且/或b)你可以控制这两者之间的联系。

But doing a proper flush is surprisingly hard. Using SO_LINGER is apparently not the way to go; see for example:

但是，适当的冲洗是非常困难的。使用SO_LINGER显然不是解决问题的方法;例如:

http://msdn.microsoft.com/en-us/library/ms740481%28v=vs.85%29.aspx
http://msdn.microsoft.com/en-us/library/ms740481%28v=vs.85%29.aspx
https://www.google.ca/#q=the-ultimate-so_linger-page
https://www.google.ca/ q = the-ultimate-so_linger-page

And SIOCOUTQ appears to be Linux-specific.

SIOCOUTQ似乎是linux特有的。

Note shutdown(fd, SHUT_WR) doesn't stop writing, contrary to its name, and maybe contrary to man 2 shutdown.

注意关闭(fd, SHUT_WR)不停止写入，与它的名称相反，可能与man 2关闭相反。

This code flushSocketBeforeClose() waits until a read of zero bytes, or until the timer expires. The function haveInput() is a simple wrapper for select(2), and is set to block for up to 1/100th of a second.

这个代码flushSocketBeforeClose()等待直到读取零字节，或者直到计时器过期。函数有input()是select(2)的简单包装器，它将被设置为1/100秒。

bool haveInput(int fd, double timeout) {
   int status;
   fd_set fds;
   struct timeval tv;
   FD_ZERO(&fds);
   FD_SET(fd, &fds);
   tv.tv_sec  = (long)timeout; // cast needed for C++
   tv.tv_usec = (long)((timeout - tv.tv_sec) * 1000000); // 'suseconds_t'

   while (1) {
      if (!(status = select(fd + 1, &fds, 0, 0, &tv)))
         return FALSE;
      else if (status > 0 && FD_ISSET(fd, &fds))
         return TRUE;
      else if (status > 0)
         FatalError("I am confused");
      else if (errno != EINTR)
         FatalError("select"); // tbd EBADF: man page "an error has occurred"
   }
}

bool flushSocketBeforeClose(int fd, double timeout) {
   const double start = getWallTimeEpoch();
   char discard[99];
   ASSERT(SHUT_WR == 1);
   if (shutdown(fd, 1) != -1)
      while (getWallTimeEpoch() < start + timeout)
         while (haveInput(fd, 0.01)) // can block for 0.01 secs
            if (!read(fd, discard, sizeof discard))
               return TRUE; // success!
   return FALSE;
}

Example of use:

使用的例子:

   if (!flushSocketBeforeClose(fd, 2.0)) // can block for 2s
       printf("Warning: Cannot gracefully close socket\n");
   closeSocket(fd);

In the above, my getWallTimeEpoch() is similar to time(), and Perror() is a wrapper for perror().

在上面，我的getWallTimeEpoch()类似于time()， Perror()是Perror()的包装器。

Edit: Some comments:

编辑:一些评论:

My first admission is a bit embarrassing. The OP and Nemo challenged the need to clear the internal so_error before close, but I cannot now find any reference for this. The system in question was HPUX 10.20. After a failed connect(), just calling close() did not release the file descriptor, because the system wished to deliver an outstanding error to me. But I, like most people, never bothered to check the return value of close. So I eventually ran out of file descriptors (ulimit -n), which finally got my attention.

我的第一次承认有点尴尬。OP和Nemo挑战了在关闭之前清除内部so_error的需要，但是现在我不能找到任何关于这个的引用。该系统的问题是hpux10.20。在连接失败之后()，只调用close()没有释放文件描述符，因为系统希望向我交付一个未完成的错误。但是，我和大多数人一样，从不费心去检查close的返回值。最后，我耗尽了文件描述符(ulimit -n)，最终引起了我的注意。
(very minor point) One commentator objected to the hard-coded numerical arguments to shutdown(), rather than e.g. SHUT_WR for 1. The simplest answer is that Windows uses different #defines/enums e.g. SD_SEND. And many other writers (e.g. Beej) use constants, as do many legacy systems.

(非常小的一点)一位解说员反对用硬编码的数字参数来关闭()，而不是像SHUT_WR(1)那样。最简单的答案是Windows使用不同的#define /enums，例如SD_SEND。还有许多其他的作家(如Beej)使用常量，就像许多遗留系统一样。
Also, I always, always, set FD_CLOEXEC on all my sockets, since in my applications I never want them passed to a child and, more importantly, I don't want a hung child to impact me.

而且，我总是在所有的套接字上设置FD_CLOEXEC，因为在我的应用程序中，我从不希望它们传递给一个孩子，更重要的是，我不想让一个挂着的孩子影响我。

Sample code to set CLOEXEC:

设置CLOEXEC的示例代码:

   static void setFD_CLOEXEC(int fd) {
      int status = fcntl(fd, F_GETFD, 0);
      if (status >= 0)
         status = fcntl(fd, F_SETFD, status | FD_CLOEXEC);
      if (status < 0)
         Perror("Error getting/setting socket FD_CLOEXEC flags");
   }

#2

Great answer from Joseph Quinsey. I have comments on the haveInput function. Wondering how likely it is that select returns an fd you did not include in your set. This would be a major OS bug IMHO. That's the kind of thing I would check if I wrote unit tests for the select function, not in an ordinary app.

约瑟夫·昆西的精彩回答。我有关于有输入功能的评论。想知道选择返回一个fd的可能性有多大，你没有包括在你的集合中。这将是一个主要的OS bug IMHO。这是我要检查的东西，如果我为select函数写单元测试，而不是在一个普通的应用程序中。

if (!(status = select(fd + 1, &fds, 0, 0, &tv)))
   return FALSE;
else if (status > 0 && FD_ISSET(fd, &fds))
   return TRUE;
else if (status > 0)
   FatalError("I am confused"); // <--- fd unknown to function

My other comment pertains to the handling of EINTR. In theory, you could get stuck in an infinite loop if select kept returning EINTR, as this error lets the loop start over. Given the very short timeout (0.01), it appears highly unlikely to happen. However, I think the appropriate way of dealing with this would be to return errors to the caller (flushSocketBeforeClose). The caller can keep calling haveInput has long as its timeout hasn't expired, and declare failure for other errors.

我的另一个评论与EINTR的处理有关。理论上，如果选择持续返回EINTR，您可能会陷入无限循环，因为这个错误让循环重新开始。考虑到非常短的超时(0.01)，看起来很不可能发生。但是，我认为处理这一问题的适当方法是将错误返回给调用者(flushSocketBeforeClose)。调用者可以一直调用有输入，因为它的超时没有过期，并声明其他错误的失败。

ADDITION #1

除了# 1

flushSocketBeforeClose will not exit quickly in case of read returning an error. It will keep looping until the timeout expires. You can't rely on the select inside haveInput to anticipate all errors. read has errors of its own (ex: EIO).

如果读取返回错误，flushSocketBeforeClose将不会快速退出。它将继续循环直到超时过期。你不能依靠内部的选择来预测所有的错误。阅读有自己的错误(例如:EIO)。

     while (haveInput(fd, 0.01)) 
        if (!read(fd, discard, sizeof discard)) <-- -1 does not end loop
           return TRUE;

#3

This sounds to me like a bug in your Linux distribution.

这对我来说就像Linux发行版中的一个bug。

The GNU C library documentation says:

GNU C图书馆文档说明:

When you have finished using a socket, you can simply close its file descriptor with close

当您使用一个套接字完成后，您可以简单地关闭它的文件描述符。

Nothing about clearing any error flags or waiting for the data to be flushed or any such thing.

清除任何错误标志或等待被刷新的数据或任何此类事件。

Your code is fine; your O/S has a bug.

你的代码很好;你的O/S有漏洞。

#1