Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have an aggregate site on a linux server that pulls in feeds from a universe of about 2,000 blogs. It's in Wordpress 3.4.2 and I have a cron job that is staggered to run five times an hour on another server to pull in the stories and then publish them to the front page of this site. This is so I didn't put too much pressure all on one server. However, the Google bots, which visit a few times every hour bring the server to its knees in the morning and evenings when there is an increase in traffic on the site. The bots have something like 30,000 links to follow at this point. How do I throttle the bots to simply grab the new stories off the front page and stop there?

EDIT- Details of my server configuration:

The way we have this set up is the server that handles all the publishing is an unmanaged instance via AWS. It mounts the NFS server and connects to the RDS to update content, etc. You get to this publishing instance via a plugin that detects the wp-admin link and then redirects you into there. The front end app server also mounts the NFS and requests data from the RDS. It is the only one that has the WP Super Cache on it.... The OS is Ubuntu on the App server and the NFS runs CentOs. The front end is Nginx and the publishing server is Apache.

share|improve this question

migrated from serverfault.com Oct 16 '12 at 2:13

This question came from our site for professional system and network administrators.

1  
Do you have any caching plugins enabled? If Googlebots are taking you down when you're under heavy load, I think you'd be better off looking at the big picture of general performance. –  silmaril8n Oct 15 '12 at 22:21
    
We have WP Super Cache on the front end server. –  Lynn Oct 15 '12 at 22:56
    
@Lynn Just a head's up, you might want to consider accepting some answers that worked on your previous questions. Users with low acceptance ratings tend not to get answers from higher-rep users. –  Magellan Oct 15 '12 at 23:24
    
Ok. I did just that. Thank you. –  Lynn Oct 15 '12 at 23:45

4 Answers 4

That is what robots.txt is for: http://www.robotstxt.org/

share|improve this answer
2  
Not really. That's for telling crawlers what not to index, but it doesn't specify throttling -- which is what the OP it looking for. –  silmaril8n Oct 15 '12 at 22:20
    
@silmaril8n: The OP asked for a method to stop the bot from descending the links from the start page. You can do this with a robots.txt –  SvenW Oct 15 '12 at 22:23
    
Ok, so I should be looking for examples of robots.txt that only sticks to the front page. –  Lynn Oct 15 '12 at 22:35
2  
Stand even further corrected. Robots.txt can be used for throttling; tell google (and everyone else) to visit at the slowest rate possible! 'Crawl-delay: 30' en.wikipedia.org/wiki/… –  user16081 Oct 15 '12 at 23:03
1  
30 seconds? 86400 / 30 is 2880, the maximum number of pages a bot could crawl in a day if it obeys that directive. It could work, but you're probably going to end up with a lot of un-indexed pages. And there's more than one bot out there, so it doesn't solve the underlying performance problem. –  Michael Hampton Oct 15 '12 at 23:07

It looks like you have a significant performance problem. Somehow I doubt that you want bots to not index your content, so I'm just not going to go there...

Absolutely the first thing you should do is set up caching. W3 Total Cache is a good start. I used to use WP Super Cache, but the former has many more options for setting up caching in various different environments. For instance, it is compatible with Amazon ElastiCache (memcached) and you will probably need to cache this way if you run multiple app servers (see below).

Using PHP-APC is another very good idea as that will reduce your CPU usage.

Beyond that, the obvious bottlenecks in your server configuration are:

  • NFS. You didn't provide any details about this, but since you're on EC2, it's probably asking for trouble.
  • Having a single front-end server. It sounds like your traffic is high enough that you've reached the limit of what a single instance can provide. It's almost certainly time to scale it up, with a larger instance, or out, with more instances behind an Elastic Load Balancer.
share|improve this answer
    
We are using WP Super Cache. We also have APC with PHP-FPM. –  Lynn Oct 15 '12 at 22:34
    
So the way we have this set up is the server that handles all the publishing is an unmanaged instance via AWS. It mounts the NFS server and connects to the RDS to update content, etc. You get to this publishing instance via a plugin that detects the wp-admin link and then redirects you into there. The front end app server also mounts the NFS and requests data from the RDS. It is the only one that has the WP Super Cache on it.... The OS is Ubuntu on the App server and the NFS runs CentOs. The front end is Nginx and the publishing server is Apache. –  Lynn Oct 15 '12 at 22:49

Use the Google Webmaster Tools and turn down the Google Crawl Frequency.

Sign in to Webmaster Tools > Configuration > Crawl Rate

share|improve this answer
    
This is the only method that works for Google and Yandex. They do not recognize the crawl-delay directive and prefer you sign up for their respective webmaster tools accounts and set the throttling through there. MSNSearch/Live/Bing has a similar function on their tools account. Use of all these is prefferable as you can do traffic shaping on the search engines so they are allowed to hit your sites in the off hours while throttling them back when the most customers/users are online viewing your website. –  Fiasco Labs Oct 16 '12 at 0:18

One way to do this is to go to the source and register at Google Webmaster's tools https://www.google.com/webmasters/tools/home?hl=en

Then once you register the site, go to the Configuration menu for your site and to Settings, there you can limit the crawl rate.

share|improve this answer
    
I did not know that. That's helpful. Thanks. –  Lynn Oct 15 '12 at 23:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.