I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html).

With htop opened, I am trying navigate the website and the much load of the website is done by the ajax link being placed on the left side of the website.

The website is hosted by a VPS with 3 proc and 2GB RAM, with enough hard with disk space.

The real problem is that this website is new and not visited much.

From the http-status module I am seeing that the overhead is caused by bots (Google bots, Bing bots, hrefs checker and so on).

So I thought that's probably due to those spiders trying to crawl all those links at once - could this be causing this overhead?

I have also put rel="nofollow" in those links, but this doesn't keep the bots away.

Is there any way through code or Plesk to disable those links to those bots?

  • Have you registered your site on Webmaster Tools? You can suggest a rate of crawling to the bots. – Prasad Ajinkya Aug 1 '13 at 11:30
  • I have registered only to google webmaster tools. But what about other? Those links are just to filter prods, I want them to be hidden to any crawler, because they dont' add any information for indexing Thanks – giuseppe Aug 1 '13 at 12:02
  • Do so for Bing Webmaster as well. If the rest of the bots are not search engines, then you might want to check if your content is being scraped using copyscape. – Prasad Ajinkya Aug 2 '13 at 4:52
  • I think you need to get memory hog protection if bots are not responding to "no follow" in .htaccess. If you don't deal with memory hog, your client's site will go down because logs are using up the storage and real traffic cannot access site. And you'll need to reinstall the vps or get a new vps. – user36394 Feb 24 '14 at 0:21

The overhead is likely being caused by the data and how it's being served:

Running a diagnostic, like here, indicated that each request for this page results in 150 separate requests, with over 2.2MB in page size, taking up to 9 seconds to load. Looking at your headers here, it appears you have no-cache specified in Cache-Control and Pragma.

You might want to enable HTTP cache and also use gzip compression with your Nginx server (as indicated in the headers as your server). See this for more: Setting up HTTP cache and gzip with nginx Also see this Google article: How gzip compression works

You may also want to check your server's core module configuration to make sure that keepalive_disable is set to none, and keepalive_requests is set to at least the default number (100). For Apache in Plesk, this thread will help with that.

Monitoring your system's resources after these modifications should indicate whether your VPS configuration is sufficient or should be upgraded.

If you still want to prevent robots from crawling the links, specify them as disallowed within your robots.txt file, as covered here.

  • You're welcome - hope it helped :-) – dan Aug 2 '13 at 7:54
  • My answer was broken :D was thanks for the question I am trying to do what are you saying. Unfortunately anyway when I put the address of category on pingdom, it makes me see nothing. Is there any other tool apart registering, to profile links on the fly or only the homepage? Thanks again – giuseppe Aug 2 '13 at 11:54
  • Pingdom Tools works, it might have just just been busy. Here is the test results for the URL above. You can enter different URL's in there, and select other cities to test from by clicking on Settings - Cheers. – dan Aug 3 '13 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.