芹菜任务的实时进度跟踪

时间：2022-01-11 23:59:18

I have a main celery task that starts multiple sub-tasks (thousands) doing multiple actions (same actions per sub-task).

我有一个主芹菜任务,它启动多个子任务(数千个)执行多个操作(每个子任务的操作相同)。

What i want is, from the main celery task to track in real-time for each action, how many are done and how many have failed for each sub-task.

我想要的是,从主芹菜任务中实时跟踪每个动作,完成了多少次以及每个子任务失败了多少次。

In summary!

  • Main task: receive list of objects, and a list of actions to do for each object.

    • 主要任务:接收对象列表以及要为每个对象执行的操作列表。

  • For each object, a sub-task is started to perform the actions for the object.

    • 对于每个对象,启动子任务以执行对象的操作。

  • The main task is finished when all the sub-tasks are finished

    • 所有子任务完成后,主要任务完成

So i need to know from the main task the real-time progress of the sub-tasks.

所以我需要从主要任务中了解子任务的实时进度。

The app i am developing is using django/angularJs, and i need to show the real-time progress asynchronously in the front-end.

我正在开发的应用程序是使用django / angularJs,我需要在前端异步显示实时进度。

I am new to celery, and i am confused and don't know how to implement this.

我是芹菜的新手,我很困惑,不知道如何实现这一点。

Any help would be appreciated. Thanks in advance.

任何帮助,将不胜感激。提前致谢。

1 个解决方案

#1

4  

I have done this before, there's too much code to put in here, so please allow me to simply put the outline, as I trust you can take care of the actual implementation and configuration:

我以前做过这个,这里有太多的代码,所以请允许我简单地提出大纲,因为我相信你可以处理实际的实现和配置:

Socket.io-based microservice to send real time events to browser

First, Django is synchronous, so it's not easy doing anything real time with it.

首先,Django是同步的,因此使用它进行任何实时操作并不容易。

So I resorted to a socket.io process. You could say it's a microservice that only listens to a "channel" that was Redis-backed, and sends notifications to a browser client that listens to a given channel.

所以我使用了socket.io进程。你可以说它只是一个微服务,它只监听Redis支持的“通道”,并向监听给定通道的浏览器客户端发送通知。

Celery -> Redis -> Socket.io -> Browser

I made it so each channel is identified with a Celery task ID. So when I fire a celery task from browser, I get the task ID, keep it and start listening to events from socket.io via that channel.

我做了这个,所以每个频道都有一个Celery任务ID。因此,当我从浏览器触发芹菜任务时,我获取任务ID,保留它并开始通过该通道从socket.io中侦听事件。

In chronological order it looks like this:

按时间顺序,它看起来像这样:

  • Fire off the Celery task, get the ID

    • 解雇Celery任务,获取ID

  • Keep the ID in your client app, open a socket.io channel to listen for updates

    • 将ID保留在客户端应用程序中,打开socket.io频道以侦听更新

  • The celery task sends messages to Redis, this will trigger socket.io events

    • celery任务向Redis发送消息,这将触发socket.io事件

  • Socket.io relays the messages to the browser, in real time

    • Socket.io实时将消息中继到浏览器

Reporting the progress

As for the actual updating of the status of the task, I just make it so that the Celery task, within its code, sends a message on Redis with something like e.g. {'done': 2, 'total_to_be_done': 10} (to represent a task that went through 2 out of 10 steps, a 20% progress, I prefer to send both numbers for better UI/UX)

至于任务状态的实际更新,我只是这样做,以便Celery任务在其代码中,在Redis上发送一条消息,例如: {'done':2,'total_to_be_done':10}(代表经历了10个步骤中的2个,20%进展的任务,我更喜欢发送两个数字以获得更好的UI / UX)

import redis
redis_pub = redis.StrictRedis()
channel = 'task:<task_id>:progress'
redis_pub.publish(channel, json.dumps({'done': 2, 'total_to_be_done': 10}))

Find documentation for publishing messages on Redis with Python here

在此处查找使用Python在Redis上发布消息的文档

AngularJS/Socket.io integration

You can use or at least get some inspiration from a library like angular-socket-io

您可以使用或至少从像angular-socket-io这样的库中获得灵感

#1

4  

I have done this before, there's too much code to put in here, so please allow me to simply put the outline, as I trust you can take care of the actual implementation and configuration:

我以前做过这个,这里有太多的代码,所以请允许我简单地提出大纲,因为我相信你可以处理实际的实现和配置:

Socket.io-based microservice to send real time events to browser

First, Django is synchronous, so it's not easy doing anything real time with it.

首先,Django是同步的,因此使用它进行任何实时操作并不容易。

So I resorted to a socket.io process. You could say it's a microservice that only listens to a "channel" that was Redis-backed, and sends notifications to a browser client that listens to a given channel.

所以我使用了socket.io进程。你可以说它只是一个微服务,它只监听Redis支持的“通道”,并向监听给定通道的浏览器客户端发送通知。

Celery -> Redis -> Socket.io -> Browser

I made it so each channel is identified with a Celery task ID. So when I fire a celery task from browser, I get the task ID, keep it and start listening to events from socket.io via that channel.

我做了这个,所以每个频道都有一个Celery任务ID。因此,当我从浏览器触发芹菜任务时,我获取任务ID,保留它并开始通过该通道从socket.io中侦听事件。

In chronological order it looks like this:

按时间顺序,它看起来像这样:

  • Fire off the Celery task, get the ID

    • 解雇Celery任务,获取ID

  • Keep the ID in your client app, open a socket.io channel to listen for updates

    • 将ID保留在客户端应用程序中,打开socket.io频道以侦听更新

  • The celery task sends messages to Redis, this will trigger socket.io events

    • celery任务向Redis发送消息,这将触发socket.io事件

  • Socket.io relays the messages to the browser, in real time

    • Socket.io实时将消息中继到浏览器

Reporting the progress

As for the actual updating of the status of the task, I just make it so that the Celery task, within its code, sends a message on Redis with something like e.g. {'done': 2, 'total_to_be_done': 10} (to represent a task that went through 2 out of 10 steps, a 20% progress, I prefer to send both numbers for better UI/UX)

至于任务状态的实际更新,我只是这样做,以便Celery任务在其代码中,在Redis上发送一条消息,例如: {'done':2,'total_to_be_done':10}(代表经历了10个步骤中的2个,20%进展的任务,我更喜欢发送两个数字以获得更好的UI / UX)

import redis
redis_pub = redis.StrictRedis()
channel = 'task:<task_id>:progress'
redis_pub.publish(channel, json.dumps({'done': 2, 'total_to_be_done': 10}))

Find documentation for publishing messages on Redis with Python here

在此处查找使用Python在Redis上发布消息的文档

AngularJS/Socket.io integration

You can use or at least get some inspiration from a library like angular-socket-io

您可以使用或至少从像angular-socket-io这样的库中获得灵感

标签：

相关文章