Django-celery和RabbitMQ没有执行任务

We've got a Django 1.3 application with django-celery 2.5.5 that's been running fine in production for month, but all of the sudden one of the celery tasks are failing to execute now.

我们有一个带有django-celery 2.5.5的Django 1.3应用程序，它已经在生产中运行了一个月，但突然之一的芹菜任务现在都无法执行。

The RabbitMQ broker and Celery workers are running on a separate machine and celeryconfig.py is configured to use that particular RabbitMQ instance as the backend.

RabbitMQ代理和Celery工作程序在单独的机器上运行，celeryconfig.py配置为使用该特定RabbitMQ实例作为后端。

On the application server I tried to manually fire off the celery task via python manage.py shell.

在应用程序服务器上，我尝试通过python manage.py shell手动触发celery任务。

The actual task is called like so:

实际任务的调用方式如下：

>>> result = tasks.runCodeGeneration.delay(code_generation, None)
>>> result
<AsyncResult: 853daa7b-8be5-4a25-a1d0-1552b38a0d21>
>>> result.state
'PENDING'

It returns an AsyncResult as expected but its status is forever 'PENDING'.

它按预期返回AsyncResult，但其状态永远是'PENDING'。

To see if the RabbitMQ broker received the message I ran the following:

要查看RabbitMQ代理是否收到消息，我运行了以下命令：

$ rabbitmqctl list_queues name messages messages_ready messages_unacknowledged | grep 853daa
853daa7b8be54a25a1d01552b38a0d21        0       0       0

I'm not sure what this means, RabbitMQ certainly seems to receive some sort of request, otherwise how else could a queue have been created for the task with id: 853daa7b8be54a25a1d01552b38a0d21. It just doesn't seem to hold any messages?

我不确定这意味着什么，RabbitMQ肯定会收到某种请求，否则如何为id为853daa7b8be54a25a1d01552b38a0d21的任务创建一个队列。它似乎没有任何消息？

I've tried restarting both Celery and RabbitMQ and the problem persists.

我已经尝试重启Celery和RabbitMQ，问题仍然存在。

Celery is run like so: $ python /home/[project]/console/manage.py celeryd -B -c2 --loglevel=INFO

Celery运行如下：$ python /home/[project]/console/manage.py celeryd -B -c2 --loglevel = INFO

Note that the celerybeat/scheduled tasks seem to be running just fine.

请注意，celerybeat /计划任务似乎运行得很好。

[EDIT]: There is no RabbitMQ configuration as it is being inlined by the init.d script: /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@hostname -boot /var/lib/rabbitmq/mnesia/rabbit@hostname-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@hostname.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@hostname-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@hostname"

[编辑]：没有RabbitMQ配置，因为它被init.d脚本内联：/usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 - -root / usr / lib / erlang -progname erl - -home / var / lib / rabbitmq - -noshell -noinput -sname rabbit @ hostname -boot / var / lib / rabbitmq / mnesia / rabbit @ hostname-plugins -expand / rabbit -kernel inet_default_connect_options [{nodelay，true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file，“/ var / log / rabbitmq / rabbit @ hostname.log”} -rabbit sasl_error_logger {file，“ /var/log/rabbitmq/rabbit@hostname-sasl.log“} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir”/ var / lib / rabbitmq / mnesia / rabbit @ hostname“

[EDIT2]: Here's the celeryconfig we're using for the workers. The same config is used for the producer except of course localhost is changed to the box with RabbitMQ broker on it.

[EDIT2]：这是我们为工人们使用的celeryconfig。生产者使用相同的配置，当然localhost更改为带有RabbitMQ代理的盒子。

from datetime import timedelta

BROKER_HOST = "localhost"

BROKER_PORT = 5672
BROKER_USER = "console"
BROKER_PASSWORD = "console"

BROKER_VHOST = "console"
BROKER_URL = "amqp://guest:guest@localhost:5672//"

CELERY_RESULT_BACKEND = "amqp"

CELERY_IMPORTS = ("tasks", )

CELERYD_HIJACK_ROOT_LOGGER = True
CELERYD_LOG_FORMAT = "[%(asctime)s: %(levelname)s/%(processName)s/%(name)s] %(message)s"


CELERYBEAT_SCHEDULE = {
    "runs-every-60-seconds": {
        "task": "tasks.runMapReduceQueries",
        "schedule": timedelta(seconds=60),
        "args": ()
    },
}

[EDIT3]: Our infrastructure is set up like number 2 below: Django-celery和RabbitMQ没有执行任务

[编辑方式3]：我们的基础设施设置如下：

1 个解决方案

#1

We solved the problem.

我们解决了这个问题。

There was a long-running celerybeat task (~430s) that was being scheduled to run every 60 seconds. This hogged all the workers in a queue.

有一个长期运行的celerybeat任务（~430秒）被安排每60秒运行一次。这困扰了队列中的所有工人。

#1