Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I would like to somehow be able to block all search engines except Google, Yahoo & Bing (and their related sites like Google Images) from crawling my site as they consume a lot of server and bandwidth but don't bring any traffic.

Is this easily done or difficult? It would be good if someone maintained a list of small search engines that could be pasted into a robots.txt file to block them.

Also, I realize I cannot block crawlers that ignore the robots.txt or sites from surreptitiously scraping and crawling, but that is not what I want. I just want to block all the Altavistas, Hotbots, Lycos (do these even still exist) and the university experiment crawlers from wasting my time.

share|improve this question
3  
Google was once a "university experiment crawler" at the time when Altavista was king... –  Dan Diplo Jul 28 '10 at 8:31
    
@Dan, meaningless, for every Google, there are 1000 university bandwidth suckers. And if any of them ever get big then they would be whitelisted. –  soupagain Sep 16 '12 at 9:08
add comment

3 Answers 3

What have you tried so far?

Using the webmaster tools robots.txt generator I made this:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

But I haven't tested it.

share|improve this answer
    
I haven't really tried anything yet until I have more info. I don't want to accidentally block Google and lose all my traffic. –  Craig Jul 28 '10 at 7:32
3  
Webmaster tools has a testing facility too. –  delete Jul 28 '10 at 8:39
add comment

How big of an issue is it really?

The bots you should be concerned about are the bots that don't follow the rules and who pretend to be regular visitors.

Search Engine traffic is legit and as Dan pointed out Google also started as a small university project. It isn't really fair to discriminate against the small guys, and possibly not smart in the long run.

Kinopiko's answer will work, and Google's webmaster tools will let you create and test your robot.txt (Site configuration, Crawler Access), but I think that if traffic from genuine search engines is a problem for you, it may be that your current hosting solution is not a good deal.

share|improve this answer
    
"It isn't really fair to discriminate against the small guys". This is ludicrous advice, webmasters are running a business, not a social service for 1000s of worthless site scrapers. –  soupagain Sep 16 '12 at 9:11
add comment

For the ones that don't follow the rules you can try to find them in your logs and then block them by IP.

Generally you can spot a bot by the fact that it reads the pages too fast to be human.

share|improve this answer
    
Or it reads your pages too regularly (e.g. exactly every 30 seconds) to be human. –  DisgruntledGoat Jul 28 '10 at 14:34
    
Except there are now 8353 site scrapers out there. You'd need to hire 5 full-time employees to process all the IP addresses. –  soupagain Sep 16 '12 at 9:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.