Hot answers tagged web-crawlers
Google only cares that they see the same thing your users see. If you're only serving up different content to these bad crawlers, and your users are getting your normal content and Google is getting that same content, you're ok.
Here is Google's document about cloaking. It defines cloaking as "the practice of presenting different content or URLs to human users and search engines." That page also has a Matt Cutts video. In the video at 0:28, Matt defines cloaking as "showing different content to users than Googlebot." Google only cares what you show to real users and what you ...
The current GoogleBot Smartphone agent, as tested with the 'Fetch as Google' Tool is essentially a fake iPhone using a headless Webkit Safari 6.0 Engine, running on a Linux x86_64 desktop machine. The default non-responsive viewport width is that of an iPhone at 980px. With a viewport <meta name="viewport" content="width=device-width, ...
You should check the site to confirm if the errors are happening or not. Also look at the server logs to see if something happened at the time Google was crawling the site and then resolved itself. It could be an intermitted error that resolved itself but might show up again. So steps to take: Confirm if errors exist on the site for the pages reported in ...
Only top voted, non community-wiki answers of a minimum length are eligible