We have a classified ad website.

Some fraudsters, scammers and kids crawl the content.

Some can be:

  • aggressive, running several EC2 with multi-threaded curl requests resulting in 100'000s requests per hour.
  • unfair, they will use the content for their website.
  • dangerous, they will use the content to spam (email or SMS) the users.
  • very dangerous, they will try to scam the users.

To protect our users against them, we would to do some cloaking (only with these limited and unwanted crawlers): modify the phone number, scramble the text, etc.

Is Google tolerant with this kind of cloaking?

  • Do the scammers pretend to be googlebot in user-agent? – Volker Siegel Jul 10 '14 at 8:36

Google only cares that they see the same thing your users see. If you're only serving up different content to these bad crawlers, and your users are getting your normal content and Google is getting that same content, you're ok.

  • Yes but the real question is, how can you identify the bad crawlers from Googlebot? As I understand, Googlebot does not always identify itself as Googlebot, so you run the risk of getting caught cloaking. – CaptainCodeman Jul 10 '14 at 8:30
  • @CaptainCodeman that is exactly the point. Google must certainly hide its identity to check if a website is cloaking. Google will not try to understand the reasons behind the difference of content... – Toto Jul 10 '14 at 11:25
  • If the anonymous-googlebot is not in your blocklist, that wont be a problem – Martijn Jul 10 '14 at 11:37

Here is Google's document about cloaking. It defines cloaking as "the practice of presenting different content or URLs to human users and search engines."

That page also has a Matt Cutts video. In the video at 0:28, Matt defines cloaking as "showing different content to users than Googlebot."

Google only cares what you show to real users and what you show to their search engine crawler. You can show different content to other user agents and Google doesn't care. You can also show different content to users that don't come from search engines and Google doesn't care. As long as the content that Googlebot sees is what users from Google will see, you won't get a penalty.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.