I'm running an Amazon EC2 'Large' instance - Ubuntu Natty x64 with PHP5 and MySQL. We execute a PHP script via CRON - this sends an email list (2000-4000 emails) using SMTP/PHPMailer.
我正在运行Amazon EC2'Warge'实例 - 使用PHP5和MySQL的Ubuntu Natty x64。我们通过CRON执行PHP脚本 - 这使用SMTP / PHPMailer发送电子邮件列表(2000-4000封电子邮件)。
The server runs very slowly (several of these CRON jobs run in parallel) and it's making the CPU go to 100%. Memory usage is low (only ~600mb / 8gigs used) and each CRON job takes a significant chunk of CPU%, for example 20-30% each with 4-5 running in parallel.
服务器运行速度非常慢(其中几个CRON作业并行运行),它使CPU达到100%。内存使用率很低(仅使用600mb / 8gig),每个CRON作业占用很大一部分CPU%,例如20-30%,每个并行运行4-5。
Trying to pinpoint the issue, I ran slow query log in MySQL but nothing caught my attention. How should I go about narrowing down the cause of this CPU usage? Is SMTP/email just that CPU intensive or is it a sign that there is a programming or server issue? Thanks!
为了查明问题,我在MySQL中运行了慢查询日志,但没有引起我的注意。我该如何缩小CPU使用率的原因? SMTP /电子邮件只是CPU密集型还是表示存在编程或服务器问题?谢谢!
EDIT: The issue is resolved. There was a trivial (of course) bug that caused emails to 'grow' (some of the previous email content was being injected into the next email) - so the email pre-processing got more and more ridiculous with each subscriber. The resulting emails had hundreds/thousands of tracking images which all hit our server simultaneously when opened i.e. 'display images' in gmail. After fending off the self-inflicted DDoS attack and two days of no sleep, I am going to enjoy a bottle of Captain Morgan while contemplating various choices I've made in life.
编辑:问题已解决。有一个微不足道的(当然)错误导致电子邮件“增长”(之前的一些电子邮件内容被注入到下一封电子邮件中) - 因此每个订阅者的电子邮件预处理变得越来越荒谬。由此产生的电子邮件有数百/数千个跟踪图像,这些图像在打开时同时点击我们的服务器,即在gmail中显示“显示图像”。在抵御自我造成的DDoS攻击和两天不睡觉之后,我将在享受一瓶摩根船长的同时考虑我在生活中做出的各种选择。
3 个解决方案
#1
1
Things that can cause this (Non-exhaustive list):
导致这种情况的事情(非详尽清单):
-
Non block IO with the SMTP server.
与SMTP服务器的非阻止IO。
-
Implementation of SMTP library used in php with long strings manipulation/long files encoded every loop (remember: the protocol must be corrected formatted, and this is checked/encoded every time you call the send method by many other methods).
在PHP中使用的SMTP库的实现,每个循环编码长字符串操作/长文件(请记住:协议必须经过格式化校正,每次通过许多其他方法调用send方法时都会检查/编码)。
-
One (or more) query per mail.
每封邮件一次(或多次)查询。
Try measuring the time spent on each operation performed inside the loop.
尝试测量在循环内执行的每个操作所花费的时间。
You can use a simple $start = microtime (true) and printf (___FILE__.':'.__LINE__.": here after % 0.8f seconds\n", microtime(true) - $start); to a debug file or another profiling tool.
你可以使用一个简单的$ start = microtime(true)和printf(___ FILE __。':'.__ LINE __。“:此后%0.8f秒\ n”,microtime(true) - $ start);到调试文件或其他分析工具。
Try to reduce protocol formatting/encoding time.
尝试减少协议格式/编码时间。
Not allow more than Number of cores in your machine here instances of php scripts running simultaneously.
不允许超过机器中的核心数量,这里同时运行php脚本的实例。
#2
1
First, establish exactly which programs are taking 100% CPU.
首先,确定哪些程序正在占用100%的CPU。
If it's the PHP interpreter then there has be something wrong in your code - an SMTP client should never manage to reach 100% utilisation because a lot of the time it'll be limited in throughput by the SMTP server.
如果它是PHP解释器,那么你的代码就会出现问题 - 一个SMTP客户端永远不应该达到100%的利用率,因为很多时候SMTP服务器的吞吐量会受到限制。
#3
1
It may not be limited just to php... is the SMTP server you are connecting to on the local box? Are you running out of sockets? Are the requests blocking themselves?
它可能不仅限于php ...是你在本地盒子上连接的SMTP服务器?你的插座用完吗?请求是否阻止了自己?
Usually for things like what are are doing, a queue based approach is usually best.
通常对于正在做的事情,基于队列的方法通常是最好的。
Have you though about using a third party service for sending mail, where all you do is send a API HTTP request? There are several benefits to this, most of these services have relationships setup with mail servers so that your emails actually get to inboxes, and your SMTP server does not get blacklisted as spammy. Amazon has a service that can do this, so do others like Postmark.
您是否在使用第三方服务发送邮件,您所做的只是发送API HTTP请求?这有几个好处,大多数这些服务都与邮件服务器建立了关系设置,以便您的电子邮件实际上到达收件箱,并且您的SMTP服务器不会被列入黑名单。亚马逊有一项服务可以做到这一点,其他像邮戳。
#1
1
Things that can cause this (Non-exhaustive list):
导致这种情况的事情(非详尽清单):
-
Non block IO with the SMTP server.
与SMTP服务器的非阻止IO。
-
Implementation of SMTP library used in php with long strings manipulation/long files encoded every loop (remember: the protocol must be corrected formatted, and this is checked/encoded every time you call the send method by many other methods).
在PHP中使用的SMTP库的实现,每个循环编码长字符串操作/长文件(请记住:协议必须经过格式化校正,每次通过许多其他方法调用send方法时都会检查/编码)。
-
One (or more) query per mail.
每封邮件一次(或多次)查询。
Try measuring the time spent on each operation performed inside the loop.
尝试测量在循环内执行的每个操作所花费的时间。
You can use a simple $start = microtime (true) and printf (___FILE__.':'.__LINE__.": here after % 0.8f seconds\n", microtime(true) - $start); to a debug file or another profiling tool.
你可以使用一个简单的$ start = microtime(true)和printf(___ FILE __。':'.__ LINE __。“:此后%0.8f秒\ n”,microtime(true) - $ start);到调试文件或其他分析工具。
Try to reduce protocol formatting/encoding time.
尝试减少协议格式/编码时间。
Not allow more than Number of cores in your machine here instances of php scripts running simultaneously.
不允许超过机器中的核心数量,这里同时运行php脚本的实例。
#2
1
First, establish exactly which programs are taking 100% CPU.
首先,确定哪些程序正在占用100%的CPU。
If it's the PHP interpreter then there has be something wrong in your code - an SMTP client should never manage to reach 100% utilisation because a lot of the time it'll be limited in throughput by the SMTP server.
如果它是PHP解释器,那么你的代码就会出现问题 - 一个SMTP客户端永远不应该达到100%的利用率,因为很多时候SMTP服务器的吞吐量会受到限制。
#3
1
It may not be limited just to php... is the SMTP server you are connecting to on the local box? Are you running out of sockets? Are the requests blocking themselves?
它可能不仅限于php ...是你在本地盒子上连接的SMTP服务器?你的插座用完吗?请求是否阻止了自己?
Usually for things like what are are doing, a queue based approach is usually best.
通常对于正在做的事情,基于队列的方法通常是最好的。
Have you though about using a third party service for sending mail, where all you do is send a API HTTP request? There are several benefits to this, most of these services have relationships setup with mail servers so that your emails actually get to inboxes, and your SMTP server does not get blacklisted as spammy. Amazon has a service that can do this, so do others like Postmark.
您是否在使用第三方服务发送邮件,您所做的只是发送API HTTP请求?这有几个好处,大多数这些服务都与邮件服务器建立了关系设置,以便您的电子邮件实际上到达收件箱,并且您的SMTP服务器不会被列入黑名单。亚马逊有一项服务可以做到这一点,其他像邮戳。