Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

My company's application has a bulk email system, all custom built, used by our clients to send emails based on opt-ins. While doing performance monitoring during some particularly large email batch sends, we noticed that there appears to be an artificial barrier caused by our current sending mechanism (phpMailer) For a brief time we re-routed our mail through a third-party SMTP service, but realized that they weren't sending any faster than we were. Now that the sending onus is back on us, we're thoroughly testing in anticipation of a number of upcoming big clients.

Assuming we improve the send rate for our mailer protocol (we're considering changing altogether to SwiftMailer) I was contemplating whether or not our SMTP server might also eventually become a bottleneck. What type of throughput are you able to achieve from your SMTP server? What considerations on SMTP sending (such as authentication, packaging, etc) might I have to reconsider when doing performance tweaks?

share|improve this question
    
Should this be a community wiki? –  RandomBen Aug 4 '10 at 18:20
1  
How many emails are you able to send (per hour), before reaching this artificial barrier? –  w3d Aug 4 '10 at 19:03
    
Right now, it appears we're at about 1,000/hour. Best we can tell, the mail is leaving the server as fast as phpMailer can process and package the message. Putting in a timing output reveals that the dynamic construction (mySql queries, etc) of the message is done in milliseconds, while the phpMailer runs for 5+ seconds/message. –  bpeterson76 Aug 4 '10 at 19:19
    
how big are your emails? You're running approx 16/minute, which could be acceptable if you're sending 500Kb emails over say, a 4Mb uplink –  Mark Henderson Aug 5 '10 at 6:27
    
@Farseeker -- Email sizes are negligible. We're sending with links to images (not embedded) and html constructed through CKEditor with heavy post-processing via both custom and tidy. Size doesn't appear to be our limitation....at least not yet. –  bpeterson76 Aug 5 '10 at 15:08
add comment

2 Answers 2

up vote 3 down vote accepted

As long as you're using a non-blocking SMTP server (in case it hits a bad DNS/SMTP server) then it is a matter of how many different domains are the emails destined for and what is the bandwidth (roughly speaking because the exceptions to the rule appear at the extremes). I would suspect the latter (bandwidth) due to you seeing similar results with an external server.

Any packaging/encryption on modern hardware takes a fraction of the time needed to send an email to a slow server. If you host your own DNS servers then make sure these return fast results as the receiving SMTP server will probably want to double check your records (slowing things down further). Having two mail servers (with enough bandwidth) is normally way faster (due to the blocking on a slow receiving server) than spending the same amount of money on a better spec'ed single server.

Nasty hacks for quick wins

  1. Sort the email addresses by domain name. If you create a new database table hanging of the table where you store the email address, it can contain the foreign key, the user part and the domain part. A run to do the split on existing data and some changes in the the CRUD to update the records will achieve this. By sorting on domain name you'll allow the sending server to re-use its connection to the remote mail server (be careful your not SPAM blocked).

  2. Where the code is hard to change and you want to alternate the mail servers you can just alter the function so it accepts a reference (prefix it with an & in PHP) then you can use a $_GLOBAL variable that is changed every x seconds by a scheduler that calls a hidden PHP page.

  3. Use a local DNS cache and query the necessary MX records before the run (or at least kick off a script to run in parallel). Most caches will hold records for 24 hours (normally written as 3600 seconds). It can reduce the initial connection latency by around 100ms. If you have multiple SMTP servers and the receiver has multiple MX records you can spoof the results. So your first sending server sees:

    • MX Record 1 Priority 10
    • MX Record 2 Priority 20
    • MX Record 3 Priority 30
    • MX Record 4 Priority 40

    and your second sending server sees

    • MX Record 1 Priority 40
    • MX Record 2 Priority 10
    • MX Record 3 Priority 20
    • MX Record 4 Priority 30

    etc. This way you can increase parallel behaviour, but it erodes any gains from point 1.

If you can run a packet trace to identify the bottlenecks, it will help find the necessary quick wins.

share|improve this answer
    
+1 Great points –  bpeterson76 Aug 5 '10 at 21:53
add comment

As much as I love PHP, that is where your bottleneck is in your system. PHP just doesn't have the efficiency of other languages when it comes to text processing.
http://stackoverflow.com/questions/603163/is-perl-a-good-option-for-heavy-text-processing

Perl is going to be a better choice for processing and sending emails. A few years back I had written a simple Perl program that would send personalized emails out to our opt-ins. I was able to send about 80,000 emails in 6 hours just using Perl to create the email and submit them to the local Sendmail program. This was on a pretty standard Virtual Private Server with 2GB of RAM.

Are you submitting the jobs to the local Sendmail process or is PHPMailer using SMTP? The local Sendmail program will be faster since PHP is not having to open up any network sockets to send the email.

So in summary you should:

  1. Use Perl instead of PHP (I hate Perl too, but it's a better text processing tool)
  2. Submit the job to the local Sendmail program (that can be configured to forward the jobs to external SMTP servers if it needs to)
  3. Use an external SMTP server if the local one gets overloaded.
share|improve this answer
    
You make great points, but that PERL thing is painful to me :) We've re-tooled the mailer and have now achieved a benchmark of more than 25K/hour simply by optimizing a query of variable data. By changing from PHPMailer to Swiftmailer we've also noticed a 25% improvement in send times. Mail is now going via local sendmail process and seems good. We'll continue to monitor to see if SMTP stands in the way. –  bpeterson76 Aug 5 '10 at 21:52
    
I would suggest you to use an (hm, lets say "more advanced") MTA. Sendmail is pretty good for "normal" Servers without special needs. But in your case (for bulk mails) i would switch to Postfix because there are many options for bulk mailing. It needs maybe a bit longer to configure it, but the effort is big. –  fwaechter Aug 8 '10 at 11:04
    
Whole heartily agree, in-fact I use Postfix, but I guess was using the term Sendmail in the general sense of the term (since Postfix is a near binary replacement for Sendmail). In truth I've never run a Sendmail server... only Postfix. –  Shane Stillwell Aug 9 '10 at 19:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.