4
votes
1answer
1k views

IIS Dynamic IP Restrictions Module and Googlebot

I consider using IIS Dynamic IP Restrictions Module in order to make it hard for DoS (Denial of Service) attacks. But I am not sure how that would work out for search engine bots like Googlebot. Any ...
1
vote
2answers
99 views

Serious Googlebot crawl spike using up a ton of bandwidth

One of the sites I maintain has seen Googlebot use 23gb of bandwidth this month. This seems utterly mental to me, what can I do about it? I have reduced the crawl rate on webmaster tools, and ...
2
votes
3answers
91 views

suspicious crawler activity

I'm noticing that I get accesses 66.249.66.198 - - [01/Jul/2011:17:13:46 +0200] "GET /img/clip.incubus.torrent.phtml HTTP/1.1" 404 143 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
10
votes
1answer
609 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...