错误:无法启动新线程。

I have a site that runs with follow configuration:

我有一个运行的网站，配置如下:

Django + mod-wsgi + apache

Django + mod-wsgi + apache。

In one of user's request, I send another HTTP request to another service, and solve this by httplib library of python.

在用户的一个请求中，我向另一个服务发送另一个HTTP请求，并由python的httplib库解决这个问题。

But sometimes this service don't get answer too long, and timeout for httplib doesn't work. So I creating thread, in this thread I send request to service, and join it after 20 sec (20 sec - is a timeout of request). This is how it works:

但是有时这个服务不会得到太长时间的响应，而对httplib的超时也不起作用。因此，我创建线程，在这个线程中，我将请求发送到服务，并在20秒后加入它(20秒-是请求的超时)。这就是它的工作原理:

class HttpGetTimeOut(threading.Thread):
    def __init__(self,**kwargs):
        self.config = kwargs
        self.resp_data = None
        self.exception = None
        super(HttpGetTimeOut,self).__init__()
    def run(self):

        h = httplib.HTTPSConnection(self.config['server'])
        h.connect()
        sended_data = self.config['sended_data']
        h.putrequest("POST", self.config['path'])
        h.putheader("Content-Length", str(len(sended_data)))
        h.putheader("Content-Type", 'text/xml; charset="utf-8"')
        if 'base_auth' in self.config:
            base64string = base64.encodestring('%s:%s' % self.config['base_auth'])[:-1]
            h.putheader("Authorization", "Basic %s" % base64string)
        h.endheaders()

        try:
            h.send(sended_data)
            self.resp_data = h.getresponse()
        except httplib.HTTPException,e:
            self.exception = e
        except Exception,e:
            self.exception = e

something like this...

是这样的……

And use it by this function:

用这个函数来表示

getting = HttpGetTimeOut(**req_config)
getting.start()
getting.join(COOPERATION_TIMEOUT)
if getting.isAlive(): #maybe need some block
    getting._Thread__stop()
    raise ValueError('Timeout')
else:
    if getting.resp_data:
        r = getting.resp_data
    else:
        if getting.exception:
            raise ValueError('REquest Exception')
        else:
            raise ValueError('Undefined exception')

And all works fine, but sometime I start catching this exception:

一切都很好，但有时我却发现了一个例外:

error: can't start new thread

at the line of starting new thread:

在开始新线程的时候:

getting.start()

and the next and the final line of traceback is

下一条和最后一条线是。

File "/usr/lib/python2.5/threading.py", line 440, in start
    _start_new_thread(self.__bootstrap, ())

And the answer is: What's happen?

答案是:发生了什么?

Thank's for all, and sorry for my pure English. :)

谢谢大家，为我的纯正英语感到抱歉。:)

5 个解决方案

#1

The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

“不能启动新的线程”错误几乎可以肯定是因为您已经有太多的线程在您的python进程中运行，并且由于某种资源限制，创建新线程的请求被拒绝。

You should probably look at the number of threads you're creating; the maximum number you will be able to create will be determined by your environment, but it should be in the order of hundreds at least.

您应该查看正在创建的线程的数量;您将能够创建的最大数量将由您的环境决定，但至少应该是数百个。

It would probably be a good idea to re-think your architecture here; seeing as this is running asynchronously anyhow, perhaps you could use a pool of threads to fetch resources from another site instead of always starting up a thread for every request.

在这里重新考虑你的架构可能是个好主意;无论如何，看到这是异步运行的，也许您可以使用一个线程池来从另一个站点获取资源，而不是总是为每个请求启动一个线程。

Another improvement to consider is your use of Thread.join and Thread.stop; this would probably be better accomplished by providing a timeout value to the constructor of HTTPSConnection.

另一个需要考虑的改进是使用线程。加入和Thread.stop;通过向HTTPSConnection的构造函数提供一个超时值，这可能会更好。

#2

You are starting more threads than can be handled by your system. There is a limit to the number of threads that can be active for one process.

您启动的线程比系统处理的线程要多。对于一个进程可以活动的线程的数量是有限制的。

Your application is starting threads faster than the threads are running to completion. If you need to start many threads you need to do it in a more controlled manner I would suggest using a thread pool.

您的应用程序启动线程的速度比线程正在运行的线程要快。如果您需要启动许多线程，您需要以更可控的方式进行，我建议使用线程池。

#3

I think the best way in your case is to set socket timeout instead of spawning thread:

我认为在您的案例中最好的方法是设置套接字超时，而不是生成线程:

h = httplib.HTTPSConnection(self.config['server'], 
                            timeout=self.config['timeout'])

Also you can set global default timeout with socket.setdefaulttimeout() function.

还可以使用socket.setdefaulttimeout()函数设置全局默认超时。

Update: See answers to Is there any way to kill a Thread in Python? question (there are several quite informative) to understand why. Thread.__stop() doesn't terminate thread, but rather set internal flag so that it's considered already stopped.

更新:查看是否有任何方法可以杀死Python中的线程?要理解其中的原因(有几个非常有用的信息)。__stop()没有终止线程，而是设置了内部标志，因此它被认为已经停止了。

#4

I completely rewrite code from httplib to pycurl.

我完全重写了从httplib到pycurl的代码。

c = pycurl.Curl()
c.setopt(pycurl.FOLLOWLOCATION, 1)
c.setopt(pycurl.MAXREDIRS, 5)
c.setopt(pycurl.CONNECTTIMEOUT, CONNECTION_TIMEOUT)
c.setopt(pycurl.TIMEOUT, COOPERATION_TIMEOUT)
c.setopt(pycurl.NOSIGNAL, 1)
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.SSL_VERIFYHOST, 0)
c.setopt(pycurl.SSL_VERIFYPEER, 0)
c.setopt(pycurl.URL, "https://"+server+path)
c.setopt(pycurl.POSTFIELDS,sended_data)

b = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, b.write)

c.perform()

something like that.

就像这样。

And I testing it now. Thanks all of you for help.

我现在测试它。谢谢大家的帮助。

#5

If you are tying to set timeout why don't you use urllib2.

如果您正在绑定设置超时，为什么不使用urllib2。

#1