3

I would like to somehow be able to block all search engines except Google, Yahoo & Bing (and their related sites like Google Images) from crawling my site as they consume a lot of server and bandwidth but don't bring any traffic.

Is this easily done or difficult? It would be good if someone maintained a list of small search engines that could be pasted into a robots.txt file to block them.

Also, I realize I cannot block crawlers that ignore the robots.txt or sites from surreptitiously scraping and crawling, but that is not what I want. I just want to block all the Altavistas, Hotbots, Lycos (do these even still exist) and the university experiment crawlers from wasting my time.

Improve this question
2
  • 3
    Google was once a "university experiment crawler" at the time when Altavista was king...
    – Dan Diplo
    Jul 28, 2010 at 8:31
  • @Dan, meaningless, for every Google, there are 1000 university bandwidth suckers. And if any of them ever get big then they would be whitelisted.
    – soupagain
    Sep 16, 2012 at 9:08

3 Answers 3

Reset to default
4

How big of an issue is it really?

The bots you should be concerned about are the bots that don't follow the rules and who pretend to be regular visitors.

Search Engine traffic is legit and as Dan pointed out Google also started as a small university project. It isn't really fair to discriminate against the small guys, and possibly not smart in the long run.

Kinopiko's answer will work, and Google's webmaster tools will let you create and test your robot.txt (Site configuration, Crawler Access), but I think that if traffic from genuine search engines is a problem for you, it may be that your current hosting solution is not a good deal.

Improve this answer
1
  • "It isn't really fair to discriminate against the small guys". This is ludicrous advice, webmasters are running a business, not a social service for 1000s of worthless site scrapers.
    – soupagain
    Sep 16, 2012 at 9:11
3

What have you tried so far?

Using the webmaster tools robots.txt generator I made this:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

But I haven't tested it.

Improve this answer
2
  • I haven't really tried anything yet until I have more info. I don't want to accidentally block Google and lose all my traffic.
    – Craig
    Jul 28, 2010 at 7:32
  • 3
    Webmaster tools has a testing facility too.
    – delete
    Jul 28, 2010 at 8:39
1

For the ones that don't follow the rules you can try to find them in your logs and then block them by IP.

Generally you can spot a bot by the fact that it reads the pages too fast to be human.

Improve this answer
2
  • Or it reads your pages too regularly (e.g. exactly every 30 seconds) to be human.
    – DisgruntledGoat
    Jul 28, 2010 at 14:34
  • Except there are now 8353 site scrapers out there. You'd need to hire 5 full-time employees to process all the IP addresses.
    – soupagain
    Sep 16, 2012 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.