在节点中使用重新支持的“kue”库。js -为什么我的redis内存使用量持续增加?

In a node.js app I'm using the kue queueing library, which is backed by redis. When a job is complete I remove it from the queue. After running about 70,000 jobs overnight the redis memory usage is at approx 30MB. There were 18 failed jobs still in the database, and the queue length is currently zero - jobs are processed more quickly than they are queuing. Redis is not being used in any other way.

在一个节点。我使用的是kue排队库，它由redis支持。当一个作业完成时，我将它从队列中删除。在一夜之间运行约70,000个作业之后，redis的内存使用量大约为30MB。数据库中仍有18个作业失败，队列长度目前为零——作业的处理速度比它们排队的速度快。Redis没有以任何其他方式使用。

Any ideas why the redis memory usage keeps increasing even though I'm deleting the completed jobs? Coffeescript code:

尽管我正在删除已完成的工作，但是为什么redis内存使用量仍然在增加?Coffeescript代码:

gaemodel.update = (params) ->
  job = jobs.create "gaemodel-update", params 
  job.attempts 2
  job.save()
  job.on "complete", ->
    job.remove (err) ->
      throw err if err
      console.log 'completed job #%d', job.id

3 个解决方案

#1

When you have a memory consumption issue with a queuing system, and you are 100% positive that all the queued items have been removed from the store and do not sit into an exception/error queue, then the most probable cause is the fact the queueing rate is much higher than the dequeuing rate.

当你有一个一个排队系统,内存消耗问题,你100%肯定所有排队的项目已经从存储中删除,不坐到一个异常/错误队列,那么最可能的原因是排队率远高于退出队列率。

Redis uses a general purpose memory allocator (jemalloc, ptmalloc, tcmalloc, etc ...). These allocators do not necessarily give the memory back to the system. When some memory is freed, the allocator tends to keep it (to reuse it for a future allocation). This is especially true when many small objects are randomly allocated, which is typically the case with Redis.

Redis使用通用内存分配器(jemalloc、ptmalloc、tcmalloc等)。这些分配器不一定将内存返回给系统。当释放一些内存时，分配器倾向于保留它(以便在将来的分配中重用它)。当许多小对象被随机分配时尤其如此，这在Redis中很常见。

The consequence is a peak of memory consumption at a given point in time will cause Redis to accumulate memory and keep it. This memory is not lost, it will be reused if another peak of memory consumption occurs. But from the system point of view, memory is still assigned to Redis. For a queuing system, if you queue the items faster than you are able to dequeue them, you will have such peak in memory consumption.

其结果是，在给定时间点上的内存消耗峰值将导致Redis积累内存并保留它。这个内存不会丢失，如果出现另一个内存消耗高峰，它将被重用。但是从系统的角度来看，内存仍然被分配给Redis。对于排队系统，如果您对项目进行排队的速度超过了对它们进行排列的速度，那么内存消耗将达到这样的峰值。

My advice would be to instrument your application to fetch and log the queue length at regular time intervals to check the evolution of the number of items in the queue (and identify the peak value).

我的建议是，检查您的应用程序，以定期的时间间隔获取和记录队列长度，以检查队列中项的数量的变化(并识别峰值值)。

Updated:

更新:

I have tested a few things with kue to understand what it stores in Redis. Actually, the data structure is quite complex (a mix of strings, sets, zsets, and hashs). If you look into Redis, you will find the following:

我用kue测试了一些东西，以了解它在Redis中存储的内容。实际上，数据结构相当复杂(字符串、集合、zset和hashs的组合)。如果你调查一下Redis，你会发现:

q:job:nnn             (hash, job definition and properties)

q:search:object:nnn   (set, metaphone tokens associated to job nnn)
q:search:word:XXXXX   (set, reverse index to support job full-text indexing)

q:jobs:inactive       (zset, all the unprocessed jobs)
q:jobs:X:inactive     (zset, all the unprocessed jobs of job type X)

q:jobs:active         (zset, all the on-going jobs)
q:jobs:X:active       (zset, all the on-going jobs of job type X)

q:jobs:complete       (zset, all the completed jobs)
q:jobs:X:complete     (zset, all the completed jobs of job type X)

q:jobs:failed         (zset, all the failed jobs)
q:jobs:X:failed       (zset, all the failed jobs of job type X)

q:jobs:delayed        (zset, all the delayed jobs)
q:jobs:X:delayed      (zset, all the delayed jobs of job type X)

q:job:types           (set, all the job types)
q:jobs                (zset, all the jobs)

q:stats:work-time     (string, work time statistic)
q:ids                 (string, job id sequence)

I don't know Coffeescript at all, so I tried to reproduce the problem using plain old Javascript:

我根本不知道Coffeescript，所以我试图用旧的Javascript来重现这个问题:

var kue = require('kue'),
    jobs = kue.createQueue();

jobs.process( 'email', function(job,done) {
  console.log('Processing email '+JSON.stringify(job) )
  done();
});

function create_email(i) {
  var j = jobs.create('email', {
    title: 'This is email '+i
    , to: 'didier'
    , template: 'Bla bla bla'
  });
  j.on('complete', function() {
    console.log('complete email job #%d', j.id);
    j.remove(function(err){
      if (err) throw err;
      console.log('removed completed job #%d', j.id);
    });
  });
  j.save();
}

for ( i=0; i<5; ++i )
{
   create_email(i);
}

kue.app.listen(8080);

I ran this code, checking what remained in Redis after processing:

我运行了这段代码，检查处理后剩下的红色部分:

redis 127.0.0.1:6379> keys *
1) "q:ids"
2) "q:jobs:complete"
3) "q:jobs:email:complete"
4) "q:stats:work-time"
5) "q:job:types"
redis 127.0.0.1:6379> zrange q:jobs:complete 0 -1
1) "1"
2) "2"
3) "3"
4) "4"
5) "5"

So it seems completed jobs are kept in q:jobs:complete and q:jobs:X:complete despite the jobs have been deleted. I suggest you check the cardinality of these zsets in your own Redis instance.

因此，似乎已完成的工作被保留在q:jobs:complete和q:jobs:X:尽管已经删除了工作，但还是完成了。我建议您在自己的Redis实例中检查这些zset的基数。

My explanation is management of these zset occurs after the 'completed' event is emitted. So the jobs are correctly removed, but their ids are inserted in those zsets just after.

我的解释是，这些zset的管理发生在发出“完成”事件之后。所以作业被正确地删除了，但是他们的id被插入到那些zset中。

A workaround is to avoid relying on per-job events, but rather use the per-queue events to remove the jobs. For instance, the following modifications can be done:

解决方案是避免依赖于每个作业事件，而是使用每个队列事件来删除作业。例如，可以做以下修改:

// added this
jobs.on('job complete', function(id) {
  console.log('Job complete '+id )  
  kue.Job.get(id, function(err, job) {
     if (err) return;
     job.remove(function(err){
        if (err) throw err;
        console.log('removed completed job #%d', job.id);
     });
  });  
});

// updated that
function create_email(i) {
  var j = jobs.create('email', {
    title: 'This is email '+i
    , to: 'didier'
    , template: 'Bla bla bla'
  });
  j.save();
}

After fixing the program, the content in Redis is much better:

修正程序后，Redis中的内容要好得多:

redis 127.0.0.1:6379> keys *
1) "q:stats:work-time"
2) "q:ids"
3) "q:job:types"

You can probably use a similar strategy from Coffescript.

您可能可以使用Coffescript相似的策略。

#2

Glad to see you fixed your problem. In any case, next time you have a memory problem with Redis, your first port of call should be the "INFO" redis command. This command will tell you valuable information such as

很高兴看到你解决了你的问题。无论如何，下次遇到Redis的内存问题时，您的第一个调用端口应该是“INFO”Redis命令。这个命令将告诉您有价值的信息，例如

Memory

used_memory:3223928 used_memory_human:3.07M used_memory_rss:1916928 used_memory_peak:3512536 used_memory_peak_human:3.35M used_memory_lua:37888 mem_fragmentation_ratio:0.59

used_memory:3223928 used_memory_human:3.07M used_memory_rss:1916928 used_memory_peak:3512536 used_memory_peak_human: 33.5 m used_memory_lua:37888 mem_fragmentation_ratio:0.59。

Or

或

Keyspace

db0:keys=282,expires=27,avg_ttl=11335089640

db0:键= 282,到期= 27日avg_ttl = 11335089640

Which is very handy for understanding the status of your memory and the keyspace at any given moment.

这对于理解内存和键空间在任何给定时刻的状态非常方便。

#3

In fact the problem was with an older version of node. Upgrading to the 0.6.x chain solved mem consumption problems.

实际上，问题出在旧版本的node上。升级到0.6。x链解决了mem的消耗问题。