0

I have only one version of my web-site, and it is optimised for both the desktop and the smartphone.

I've noticed that Google crawls each page exactly twice: once with Googlebot/2.1, and again with Googlebot-Mobile/2.1:

% fgrep "/www/drupal6/filefield " /var/www/logs/ports.su/ports.su.access.log
66.249.76.223 - - [12/Mar/2013:07:46:42 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.223 - - [12/Mar/2013:13:59:23 -0700] "GET /www/drupal6/filefield HTTP/1.1" 200 2365 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

I don't think blocking either of these bots in /robots.txt would be appropriate, as it might remove my site from their respective index.

However, don't they share their indices, too?

How do I tell Google that I only have one site?

Looking for something that's the opposite of Vary: User-Agent.

Improve this question
1
  • Why does this question has a -2? Am I an idiot for expecting a solution to this problem?
    – cnst
    Commented Apr 24, 2013 at 20:32

2 Answers 2

Reset to default
3

You are already telling them because you have one site using a single set of URLs.

They visit twice because each bot evaluates differently. In the end it goes in the same index but the metadata is different which lets them serve different results to standard and mobile queries.

Improve this answer
2
  • 1
    this is not an answer — the question is pretty clear that there is only one set of URLs to begin with.
    – cnst
    Commented Mar 12, 2013 at 21:49
  • Sorry for not being clear enough, I meant this is already the case. Edited for clarity.
    – Itai
    Commented Mar 12, 2013 at 22:32
1

Having Googlebot/2.1 and Googlebot-Mobile/2.1 is a good thing, you should not block either.

It's important to note that when viewing Google search results on a mobile phone they tend to vary from a Desktop Google search since Google favors sites that are viewable on a mobile phone. Blocking Googlebot-Mobile/2.1 may prevent people finding your site when they use Google search on their phone.

Improve this answer
2
  • Yes, I agree that blocking does not sound like a solution. But with 8k individual pages, it seems like a huge waste of resources to be requesting each page twice. Note that the requests even come from the same IPv4-address over at Google. The pages are static (re-generated daily) — why is Google downloading every page twice, instead of going forward with its indexing otherwise? There should be a way to prevent this.
    – cnst
    Commented Mar 12, 2013 at 21:54
  • There is no work around for this as far as I know... In terms of Google using up your resources well you have the option of blocking them, otherwise you have to lump it... To be honest with good hosting you never have to be concerned with the usage of resources, your bandwidth and system resources should be budgeted into your daily visits which includes usage from bots otherwise you need to upgrade. The more popular your site is the more often Google will visit your site too, and again this is a good thing. You should have some kind of return of investment (ie ads) on the hosting for upgrades.
    – Simon Hayter
    Commented Mar 12, 2013 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.