[[Django Celery]] Celery阻止了IO任务

I use celery to do some IO tasks, such as grab remote image, sending email to users. But celery sometimes blocked with no logs. At this time, it won't do any task i send. I have to restart it, it begin to work where it blocked.

我使用芹菜做一些IO任务,比如抓取远程图像,向用户发送电子邮件。但芹菜有时会被堵塞,没有原木。这时,它不会执行我发送的任何任务。我必须重新启动它,它开始工作在它阻止的地方。

It puzzles me for a very long time. What can i do ? And what is the best practice for distributing IO tasks with celery?

这困扰了我很长一段时间。我能做什么 ?用芹菜分配IO任务的最佳做法是什么?

2 个解决方案

#1

By default, celery worker fork several processes waiting for tasks request from client. For the tasks of IO pending and your system need a larger number of concurrency that handle request concurrently. Here is the command:

默认情况下,celery worker会从客户端分叉几个进程等待任务请求。对于IO挂起的任务,您的系统需要大量并发处理请求并发。这是命令:

    celery -A tasks worker --without-heartbeat -P threads --concurrency=10

If simutanelous income requests is a lot, your concurrency level have to set higher than the size of incoming reqeust burst. The system's performance may be limited by the hardware memeory size or OS's select API. You can use celery's thread/ gevent model when concurrency is large:

如果同时收入请求很多,则您的并发级别必须设置为高于传入请求突发的大小。系统的性能可能受硬件内存大小或操作系统选择API的限制。当并发性很大时,你可以使用celery的thread / gevent模型:

    celery -A tasks worker --without-heartbeat -P threads --concurrency=1000

    celery -A tasks worker --without-heartbeat -P gevent --concurrency=1000

#2

you can increase the celery concurrency

你可以增加芹菜的并发性

manage.py celeryd   --concurrency=3

where concurrency == number of processors

其中concurrency ==处理器数量

run shell command

运行shell命令

   grep -c processor /proc/cpuinfo

to get number of processors on your machine

获取计算机上的处理器数量

#1