Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Question

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.

I decreased crawl rate from Google webmaster tools. But I'm supposing to use these three tools:

• Adding rel="nofollow" to my inner pages. So Googlebot won't crawl and index them.
• Adding meta tag "noindex" so Google will remove this page from index and won't get it again.
• Adding Disallow: /mySomeFolder/ to robots.txt and Googlebot won't crawl that pages.

I'm planning to use these methods for my 56.000 pages, except the most important 6-7 pages. Which method would you prefer and what would be disadvantages or advantages ? Or won't it change my website speed etc..

Answer 1

You can set the rate at which Googlebot crawls your website in Google Webmaster Tools. http://support.google.com/webmasters/bin/answer.py?hl=en&answer=48620

This is how you should limit Googlebots access not by denying it access. Because noindex pages they will still visit causing hits on your server. Robots.txt disallow may work but be careful you correctly edit that file. nofollow is only Good for one specific link. If you have a link to that page in your sitemap they'll still crawl it.

Crawl rate refers to the speed of Googlebot's requests during the crawl process. It doesn't have any effect on how often we crawl or how deeply we crawl your URL structure. If you want Google to crawl new or updated content on your site, use Fetch as Google instead.

Google has sophisticated algorithms that determine how much to crawl each site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.

If Google is crawling your site too often, and slowing down your server, you can change the crawl rate (the time used by Googlebot to crawl the site) for sites that are at the root level—for example, www.example.com and http://subdomain.example.com. Changing the crawl rate can cause some problems (for example, Google will not be able to crawl at a faster rate than the custom rate you set), so don't do this unless you are noticing specific problems caused by Googlebot accessing your servers too often.

You can't change the crawl rate for sites that are not at the root level—for example, www.example.com/folder.

Answer 2

Using any of the above mention 3 "solutions" (nofollow, noindex or Disallow) will seriously harm your SEO rankings.

You should control your crawl rate via GMTW (as suggested by @Anagio [+1])

If this still doesn't work, you may be dealing with Googlebot impersonator that uses Google's user-agent to scrape or spam your site. I think this might be the case because I never (in my 8 years as SEO) experienced any problems with Googlebot default crawl rating...

To identify impersonator I suggest using Botopedia.org IP verification feature to check if this visitor's IP is indeed connected to Google. Here is the link: Googlebot IP check

You can learn more about Googlebot impersonation phenomena here: study shows: 16% of all Googlebot visits are fake