I'm confused about Task execution using queues. I've read the documentation and I thought I understood bucket_size and rate, but when I send 20 Tasks to a queue set to 5/h, size 5, all 20 Tasks execute one after the other as quickly as possible, finishing in less than 1 minute.
我对使用队列的任务执行感到困惑。我已经阅读了文档,我认为我理解了bucket_size和rate,但是当我将20个任务发送到设置为5 / h,大小为5的队列时,所有20个任务尽可能快地执行,最后完成1分钟。
deferred.defer(spam.cookEggs,
egg_keys,
_queue="tortoise")
- name: tortoise
rate: 5/h
bucket_size: 5
What I want is whether I create 10 or 100 Tasks, I only want 5 of them to run per hour. So it would take 20 Tasks approximately 4 hours to complete. I want their execution spread out.
我想要的是我是创建10个还是100个任务,我只想要每小时运行5个任务。因此,完成约需4个小时需要20个任务。我希望他们的执行分散开来。
UPDATE
The problem was I assumed that when running locally, that Task execution rate rules were followed, but that is not the case. You cannot test execution rates locally. When I deployed to production, the rate and bucket size I had set executed as I expected.
问题是我假设在本地运行时遵循了任务执行率规则,但事实并非如此。您无法在本地测试执行率。当我部署到生产时,我设置的速率和桶大小按照我的预期执行。
2 个解决方案
#1
6
Execution rates are not honored by the app_devserver. This issue should not occur in production.
app_devserver不支持执行率。生产中不应出现此问题。
[Answer discovered by Nick Johnson and/or question author; posting here as community wiki so we have something that can get marked accepted]
[Nick Johnson和/或提问者发现的答案;在这里发布为社区维基,所以我们有一些可以被标记接受的东西]
#2
-1
You want to set bucket_size to 1, or else you'll have "bursts" of queued activity like you saw there.
您想将bucket_size设置为1,否则您将看到排队活动的“爆发”。
From the documentation:
从文档:
bucket_size
Limits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution rate. For example, consider a queue with a rate of 5/s and a bucket size of 10. If that queue has been inactive for some time (allowing its "token bucket" to fill up), and 20 tasks are suddenly enqueued, it will be allowed to execute 10 tasks immediately. But in the following second, only 5 more tasks will be able to be executed because the token bucket has been depleted and is refilling at the specified rate of 5/s.
限制队列处理的突发性,即更高的桶大小允许队列执行速率更大的峰值。例如,考虑一个速率为5 / s且桶大小为10的队列。如果该队列已处于非活动状态一段时间(允许其“令牌桶”填满),并且20个任务突然排队,它将被允许立即执行10个任务。但是在接下来的第二步中,只能再执行5个任务,因为令牌桶已经耗尽并且以指定的5 / s速率重新填充。
#1
6
Execution rates are not honored by the app_devserver. This issue should not occur in production.
app_devserver不支持执行率。生产中不应出现此问题。
[Answer discovered by Nick Johnson and/or question author; posting here as community wiki so we have something that can get marked accepted]
[Nick Johnson和/或提问者发现的答案;在这里发布为社区维基,所以我们有一些可以被标记接受的东西]
#2
-1
You want to set bucket_size to 1, or else you'll have "bursts" of queued activity like you saw there.
您想将bucket_size设置为1,否则您将看到排队活动的“爆发”。
From the documentation:
从文档:
bucket_size
Limits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution rate. For example, consider a queue with a rate of 5/s and a bucket size of 10. If that queue has been inactive for some time (allowing its "token bucket" to fill up), and 20 tasks are suddenly enqueued, it will be allowed to execute 10 tasks immediately. But in the following second, only 5 more tasks will be able to be executed because the token bucket has been depleted and is refilling at the specified rate of 5/s.
限制队列处理的突发性,即更高的桶大小允许队列执行速率更大的峰值。例如,考虑一个速率为5 / s且桶大小为10的队列。如果该队列已处于非活动状态一段时间(允许其“令牌桶”填满),并且20个任务突然排队,它将被允许立即执行10个任务。但是在接下来的第二步中,只能再执行5个任务,因为令牌桶已经耗尽并且以指定的5 / s速率重新填充。