Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have only one version of my web-site, and it is optimised for both the desktop and the smartphone.

I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with Googlebot-Mobile/2.1:

% fgrep "/www/drupal6/filefield " /var/www/logs/ports.su/ports.su.access.log
66.249.76.223 - - [12/Mar/2013:07:46:42 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.223 - - [12/Mar/2013:13:59:23 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

I don't think blocking either of these bots in /robots.txt would be appropriate, as it might remove my site from their respective index.

However, don't they share their indices, too?

How do I tell Google that I only have one site?

Looking for something that's the opposite of Vary: User-Agent.

share|improve this question
    
Why does this question has a -2? Am I an idiot for expecting a solution to this problem? –  cnst Apr 24 '13 at 20:32

2 Answers 2

You are already telling them because you have one site using a single set of URLs.

They visit twice because each bot evaluates differently. In the end it goes in the same index but the metadata is different which lets them serve different results to standard and mobile queries.

share|improve this answer
1  
this is not an answer — the question is pretty clear that there is only one set of URLs to begin with. –  cnst Mar 12 '13 at 21:49
    
Sorry for not being clear enough, I meant this is already the case. Edited for clarity. –  Itai Mar 12 '13 at 22:32

Having Googlebot/2.1 and Googlebot-Mobile/2.1 is a good thing, you should not block either.

It's important to note that when viewing Google search results on a mobile phone they tend to vary from a Desktop Google search since Google favors sites that are viewable on a mobile phone. Blocking Googlebot-Mobile/2.1 may prevent people finding your site when they use Google search on their phone.

share|improve this answer
    
Yes, I agree that blocking does not sound like a solution. But with 8k individual pages, it seems like a huge waste of resources to be requesting each page twice. Note that the requests even come from the same IPv4-address over at Google. The pages are static (re-generated daily) — why is Google downloading every page twice, instead of going forward with its indexing otherwise? There should be a way to prevent this. –  cnst Mar 12 '13 at 21:54
    
There is no work around for this as far as I know... In terms of Google using up your resources well you have the option of blocking them, otherwise you have to lump it... To be honest with good hosting you never have to be concerned with the usage of resources, your bandwidth and system resources should be budgeted into your daily visits which includes usage from bots otherwise you need to upgrade. The more popular your site is the more often Google will visit your site too, and again this is a good thing. You should have some kind of return of investment (ie ads) on the hosting for upgrades. –  bybe Mar 12 '13 at 22:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.