Tagged Questions
0
votes
0answers
263 views
blocking bad bots with robots.txt in 2012 [closed]
does it still work good? I have this:
# Generated using http://solidshellsecurity.com services
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
...
1
vote
2answers
62 views
Serious Googlebot spike
One of the sites I maintain has seen googlebot use 23gb of bandwidth this month.
This seems utterly mental to me, what can I do about it?
I have reduced the crawl rate on webmaster tools, and ...
3
votes
5answers
2k views
Disallow robots.txt from being accessed in a browser but still accessible by spiders?
We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't ...
