如何在刷新到mysql之前处理memcache中的大页面印象

I'm writing an analytics tool which will essentially log every time a page is hit. The problem I have is the amount of impressions could become very high and I will potentially need to handle millions a day or at least ~300 a second (I've head people say they can achieve 10k per second). My current setup is php logging each time a page is accessed as an insert directly into MySQL innodb table, this is becoming slow and the server appears to miss a hit occasionally which isn't too bad for analytics but not ideal. My idea is to put memcache in the front and log all impressions to that first and then run a scheduled batching process which takes the impressions from memcache and then puts them into the db at a quiet period e.g. 3am. So my question is

我正在编写一个分析工具，每次页面被点击时都会记录。我遇到的问题是印象量可能会变得非常高，我可能需要每天处理数百万或者每秒至少300次（我的头人说他们可以达到每秒10k）。我目前的设置是每次将页面作为插入直接插入MySQL innodb表时进行php日志记录，这变得很慢，并且服务器似乎偶尔会错过一次点击，这对于分析而言并不是很糟糕但不理想。我的想法是将memcache放在前面并首先记录所有的印象，然后运行预定的批处理过程，该过程从memcache获取印象，然后在安静的时间段将它们放入数据库中，例如凌晨3点。所以我的问题是

1) Is this the right way to do it? Ideally I do not want to implement any new technology or implement a NoSQL db such as mongo or couch.

1）这是正确的方法吗？理想情况下，我不想实现任何新技术或实现NoSQL数据库，如mongo或沙发。

2) How would I do this in memcache? As there is no way of finding out all of the keys currently in cache I was thinking of setting a counter which increments on each impression and then a key with the counter as an id containing the data, the batch process would loop from its last position to the counter and process those records and then set its last position to the current counter..

2）我如何在memcache中执行此操作？由于无法找到当前缓存中的所有密钥，我正在考虑设置一个计数器，该计数器在每次展示时递增，然后将计数器作为包含数据的id的键，批处理将从其最后位置循环到柜台并处理这些记录，然后将其最后位置设置为当前计数器。

Any help is appreciated

任何帮助表示赞赏

Cheers

干杯

1 个解决方案

#1

I wouldn't create an serperate batch, but instead integratie it in your code.

我不会创建一个serperate批处理，而是在你的代码中集成它。

Note that there is no checking/validation/locking here. Some pseudocode:

请注意，这里没有检查/验证/锁定。一些伪代码：

$mViews = $oMemcache->get('views');
if ($mViews > 1000) {
    $oDatabase->update($mViews);
    $oMemcached->replace('views', 1);
}
else {
    $oMemcached->increment('views');
}

#1