how to set crawl rate like the old option of webmastertools like 7days or a month

Question

My server people are blocking ip's which do a lot of requests including google bot. I found this by checking my webmastertools account. there it said network unreachable for all the urls which are indexed in google. Then i thought of reducing the rate of crawl. And i found that google will not alter the crawl rate and it is google who will determine and the users are allowed to alter the no of requests per time.

Before some months/years i have seen the crawl rate had options like days, week and months but not it is different. How to set crawl rate for the above different criteria?

Answer 1

Google Webmaster Tools > YourSite > Site configuration > Settings > Crawl rate > Set costum crawl rate.

The slowest rate possible is 500seconds between requests.

This link might help your "server people": http://googlewebmastercentral.blogspot.com/2006/09/how-to-verify-googlebot.html

Answer 2

Setup a sitemap xml file, and specify the frequency which your data changes. This wont force the bots to stop visiting your site, but it should reduce it.

Something like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>http://www.example.com/</loc>
      <lastmod>2005-01-01</lastmod>
      <changefreq>weekly</changefreq>
      <priority>0.8</priority>
   </url>
</urlset>