1
vote
2answers
98 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
3
votes
5answers
2k views

Disallow robots.txt from being accessed in a browser but still accessible by spiders?

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...