1
vote
3answers
206 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
-2
votes
2answers
267 views

How do I prevent duplicate indexing by Googlebot-Mobile?

I have only one version of my web-site, and it is optimised for both the desktop and the smartphone. I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with ...