How to make scraping of own content through wget, httrack etc. impossible, but allow crawling through googlebot?

This should be done without showing to googlebot other content, as to other user agents.

And, please, better avoid IP recognition in your advices, if this is in general possible!

In current setup it works already based on IP recognition and the server goes periodically down. The setup is like:

  • first layer: nginx as caching,
  • second layer: apache with mod_security. mod_security makes IP recognition and manages traffic,
  • third layer: tomcat with CMS).

The main bottleneck is currently mod_security, and, partly, the way from mod_security to tomcat. Setup change is outside of manifold including viable solutions.

  • Mod security is one of the very best. You are saying IP recognition is your problem but I seriously doubt it. The reason why IP addresses are used is because the rest if the request packet can easily be spoofed whereas the address is not. User agents are highly unreliable. IP addresses are extremely reliable. How do you know specifically where your problem exists? What do you find in your log files?
    – closetnoc
    Apr 26, 2017 at 15:02
  • Webmaster can change wget user-agent if you have blocked. Top website use algo to detect such a connection, and block that kind of scrapper, but scrapper comes with proxy to make another connection. I don't think there is any perfect solution.
    – Goyllo
    Apr 26, 2017 at 15:09
  • @closetnoc do you mean, ip recognition should be the only reliable solution?
    – Evgeniy
    Apr 26, 2017 at 16:25
  • So far i understand my colleague, the bottleneck begins after the amount of concurrent requests becomes higher then some hundreds, because each request should be checked, googlebot or not.
    – Evgeniy
    Apr 26, 2017 at 16:31
  • 2
    IP addresses are the only reasonablely reliable way to properly block bad requests. As well, mod security is used on huge sites without an issue. I am starting to wonder if there is not a resource issue somewhere be it memory, cpu, disk speed, etc. You may be hitting a ceiling somewhere.
    – closetnoc
    Apr 26, 2017 at 23:00

1 Answer 1


It is almost impossible to rule out third party scraping entirely. The first line of defense is a robots.txt file:

User-Agent: Googlebot

User-Agent: *
Disallow: /

That will disallow all crawlers except Googlebot that obey robots.txt.

  • true. but this applies only to bots/crawlers/scrapers which adhere to the robots.txt rules. Google, Yahoo, Bing do, but most scrapers simply ignore the robots.txt file.
    – David K.
    Jun 20, 2017 at 15:12
  • 1
    Right, I included that disclaimer. Jun 20, 2017 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.