I have only one version of my web-site, and it is optimised for both the desktop and the smartphone.
I've noticed that Google crawls each page exactly twice: once with
Googlebot/2.1, and again with
% fgrep "/www/drupal6/filefield " /var/www/logs/ports.su/ports.su.access.log 18.104.22.168 - - [12/Mar/2013:07:46:42 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 22.214.171.124 - - [12/Mar/2013:13:59:23 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
I don't think blocking either of these bots in
/robots.txt would be appropriate, as it might remove my site from their respective index.
However, don't they share their indices, too?
How do I tell Google that I only have one site?
Looking for something that's the opposite of