We have a pretty large website, which handles a couple million visitors a month, and we're currently developing an API for some of our partners. In the near future, we hope to expose the API to all of our visitors.
我们有一个非常大的网站,每月处理几百万访问者,我们目前正在为我们的一些合作伙伴开发API。在不久的将来,我们希望向所有访问者公开API。
As we are trying to limit the number of requests to say, 100.000 / day and 1.000 / minute, I need to log the total number of requests per API key, and verify that the API user doesn't exceed this total. We won't check real-time whether the limit is exceeded at this point, but afterwords in the site's control panel. We also need to display a timeline per user in the control panel, so we have to get a quick per day or per hour overview if we need this.
由于我们试图限制请求数量,即100.000 /天和1.000 /分钟,我需要记录每个API密钥的请求总数,并验证API用户是否未超过此总数。我们不会实时检查此时是否超出限制,而是检查站点控制面板中的后续内容。我们还需要在控制面板中显示每个用户的时间表,因此如果需要,我们必须快速获得每日或每小时的概述。
My first idea was to build the following app:
我的第一个想法是构建以下应用程序:
API User => Webserver => Posts message with API key to a message queue => Service picks up the message => Posts to the database, where there is 1 item for each user-hour combo (key|hour|count). Which is gonna be quite fast, yet we'll remove quite some useful information (queries, requests / minute, etc.) Saving each and every request as separate record in the database will likely generate millions of records a day, and will (I guess, I'm not that much of a DBA) be quite slow when generating some chart. Even with the correct indices.
API User => Webserver =>将带有API密钥的消息发送到消息队列=>服务接收消息=>帖子到数据库,其中每个用户小时组合有1个项目(键|小时|计数)。这将是相当快的,但我们将删除相当一些有用的信息(查询,请求/分钟等)。将每个请求作为单独的记录保存在数据库中可能每天产生数百万条记录,并且(我猜测,我不是那么多DBA)在生成一些图表时会非常慢。即使有正确的指数。
Our platform consists of around ten webservers, ten front end SQL servers, statsserver, some other servers for processing large tasks. All Windows (except our EMC), and MS SQL. Dev platform is ASP.Net WCF.
我们的平台包括大约10个Web服务器,10个前端SQL服务器,statsserver,以及一些用于处理大型任务的服务器。所有Windows(我们的EMC除外)和MS SQL。开发平台是ASP.Net WCF。
3 个解决方案
#1
My advice would be to log everything - a simple append-only text file is simplest - and have a background task periodically read and summarize log segments into the database. This has several advantages over more 'sophisticated' approaches:
我的建议是记录所有内容 - 一个简单的仅附加文本文件是最简单的 - 并且有一个后台任务定期读取并将日志段汇总到数据库中。与更“复杂”的方法相比,这有几个优点:
- It's simpler.
- It's really easy to debug.
- You can keep individual log segments around until you need to delete them from disk, so you can get information on individual requests for debugging and accounting purposes.
- You can easily extend it to collect more information, or improve and change your summarizer, because the components are loosely coupled.
- It's easy to shard - just have each server keep its own logs.
它很容易调试。
您可以保留各个日志段,直到需要从磁盘中删除它们为止,这样您就可以获得有关调试和记帐目的的各个请求的信息。
您可以轻松扩展它以收集更多信息,或者改进和更改摘要生成器,因为组件是松散耦合的。
分片很容易 - 让每个服务器都保留自己的日志。
#2
I'd start with logging at first, and leave off enforcement. Your logging may show you that you don't need enforcement, or it may show you you need a different kind of enforcement.
我首先开始记录,然后不执行。您的日志记录可能会向您显示您不需要执行,或者它可能表明您需要执行其他类型的操作。
I'd just start off creating a simple logging API: ApiLogger.Log(apiKey). I'd have the logger take authentication information etc. from HttpContext. I'd start at first just dumping it into a database table, and only get fancier if performance required it.
我刚刚开始创建一个简单的日志API:ApiLogger.Log(apiKey)。我有记录器从HttpContext获取认证信息等。我首先开始将它转储到数据库表中,只有在性能要求时才会变得更加漂亮。
Later analysis could determine who is making how many calls, whether you want multiple tiers, charging different amounts per tier, etc. But for the moment, just store the data that your Business people will need.
以后的分析可以确定谁在拨打多少电话,是否需要多层,每层收取不同的金额等等。但目前,只需存储您的业务人员需要的数据。
#3
As we are trying to limit the number of requests to say, 100.000 / day and 1.000 / minute, I need to log the total number of requests per api key, and verify that the api user doesn't exceed this total.
由于我们试图限制请求数量,即100.000 /天和1.000 /分钟,我需要记录每个api密钥的请求总数,并验证api用户不超过此总数。
A feature like this will be part of WCF (if not already) in the very near future. I am currently racking my brain on where I heard it so I can point you in the right direct.
像这样的功能将在不久的将来成为WCF的一部分(如果还没有)。我正在把我的大脑放在我听到的地方,所以我可以直接指出你。
EDIT: FOUND IT! This week on a podcast called "The Thirsty Developer", this very topic came up. Download the podcast here, and at 39:40 into the podcast the topic comes up. For those that do not want to listen there is a REST toolkit that has this feature in it. I think the toolkit can be found here
编辑:发现它!本周在一个名为“The Thirsty Developer”的播客上,出现了这个话题。在这里下载播客,并在39:40进入播客主题出现。对于那些不想听的人,有一个REST工具包,里面有这个功能。我认为可以在这里找到工具包
#1
My advice would be to log everything - a simple append-only text file is simplest - and have a background task periodically read and summarize log segments into the database. This has several advantages over more 'sophisticated' approaches:
我的建议是记录所有内容 - 一个简单的仅附加文本文件是最简单的 - 并且有一个后台任务定期读取并将日志段汇总到数据库中。与更“复杂”的方法相比,这有几个优点:
- It's simpler.
- It's really easy to debug.
- You can keep individual log segments around until you need to delete them from disk, so you can get information on individual requests for debugging and accounting purposes.
- You can easily extend it to collect more information, or improve and change your summarizer, because the components are loosely coupled.
- It's easy to shard - just have each server keep its own logs.
它很容易调试。
您可以保留各个日志段,直到需要从磁盘中删除它们为止,这样您就可以获得有关调试和记帐目的的各个请求的信息。
您可以轻松扩展它以收集更多信息,或者改进和更改摘要生成器,因为组件是松散耦合的。
分片很容易 - 让每个服务器都保留自己的日志。
#2
I'd start with logging at first, and leave off enforcement. Your logging may show you that you don't need enforcement, or it may show you you need a different kind of enforcement.
我首先开始记录,然后不执行。您的日志记录可能会向您显示您不需要执行,或者它可能表明您需要执行其他类型的操作。
I'd just start off creating a simple logging API: ApiLogger.Log(apiKey). I'd have the logger take authentication information etc. from HttpContext. I'd start at first just dumping it into a database table, and only get fancier if performance required it.
我刚刚开始创建一个简单的日志API:ApiLogger.Log(apiKey)。我有记录器从HttpContext获取认证信息等。我首先开始将它转储到数据库表中,只有在性能要求时才会变得更加漂亮。
Later analysis could determine who is making how many calls, whether you want multiple tiers, charging different amounts per tier, etc. But for the moment, just store the data that your Business people will need.
以后的分析可以确定谁在拨打多少电话,是否需要多层,每层收取不同的金额等等。但目前,只需存储您的业务人员需要的数据。
#3
As we are trying to limit the number of requests to say, 100.000 / day and 1.000 / minute, I need to log the total number of requests per api key, and verify that the api user doesn't exceed this total.
由于我们试图限制请求数量,即100.000 /天和1.000 /分钟,我需要记录每个api密钥的请求总数,并验证api用户不超过此总数。
A feature like this will be part of WCF (if not already) in the very near future. I am currently racking my brain on where I heard it so I can point you in the right direct.
像这样的功能将在不久的将来成为WCF的一部分(如果还没有)。我正在把我的大脑放在我听到的地方,所以我可以直接指出你。
EDIT: FOUND IT! This week on a podcast called "The Thirsty Developer", this very topic came up. Download the podcast here, and at 39:40 into the podcast the topic comes up. For those that do not want to listen there is a REST toolkit that has this feature in it. I think the toolkit can be found here
编辑:发现它!本周在一个名为“The Thirsty Developer”的播客上,出现了这个话题。在这里下载播客,并在39:40进入播客主题出现。对于那些不想听的人,有一个REST工具包,里面有这个功能。我认为可以在这里找到工具包