5

We are currently trying to see if our site is being indexed successfully by google. Mind you we got millions of pages.

A trick some people use is the google site: command. I always assumed that google.com and google.nl would be the same. Data being fetched by the same GoogleBot. And that it was more of a latency thing where google.nl would be closer to home than google.com. But this is clearly not the case, since google.nl and google.com give back different results.

http://www.google.nl/search?sourceid=chrome&ie=UTF-8&q=site:amazon.com returns 37.400.000 results and http://www.google.com/#sclient=psy&hl=en&site=&source=hp&q=site:amazon.com returns 487,000,000 results

Does anybody know how this works?

Another related question is what other tricks people use to see if google.com successfully indexes all data.

It seems the best way is to do some analytics on the apache logs searching for the GoogleBot.

4
  • I have always thought that the only the order is different but the results stays the same between different google domains. That is really, really strange. Nov 30, 2010 at 11:30
  • "Local" Google sites have always given different results for me. I have used and/or tried at least 4 different ones. Each local site is optimized (or so it's supposed to be) for users of that particular domain. google.co.uk for instance gives results that are more relevant for visitors from the UK. Nov 30, 2010 at 13:24
  • Muhammed: But I don't see the optimization here. I took Amazon as an example, but a Dutch site slows less results on google.nl (dutch). I can't see this being a optimization.
    – Pullie
    Nov 30, 2010 at 21:12
  • I saw once (long ago) that the Google data centres are not always sync'd, so this is a reason why there are differences between the results on different sites. However, i think the example you show here goes far beyond differences in the data stored in various data centres. There's clearly some other logic going on. Dec 1, 2010 at 1:22

1 Answer 1

1

Google Webmaster Tools is the easiest (and most authoritative) way to determine how much of your content has made it into Google's index.

5
  • Google Webmaster tools does not show how many pages are being indexed. It will say how many URL's are in the index based on the sitemap. Which in our case says 0. Something we try to discuss with Google since that is clearly not the case. And it does not explain the location difference.
    – Pullie
    Nov 30, 2010 at 12:20
  • @Ronald Pulleman - Valid points, however, the alternative to taking what Google freely provides isn't pretty. Last I checked a few years back, programs like Web CEO had difficulty with request throttling when one tried to grab search results en masse. Note that the numbers which Google returns as "total" for hits on a term are useless by Google's own admission (estimate differences by order of magnitude are not uncommon): google.com/support/webmasters/bin/…
    – danlefree
    Nov 30, 2010 at 15:29
  • danlefree: fair enough. However I also cannot find certain products in google.nl which I can find in google.com so there seems to be a difference in the index
    – Pullie
    Nov 30, 2010 at 21:38
  • @Ronald Pulleman - Can you give an example? (preferably something like "blue widget" site:yourdomain.com)
    – danlefree
    Dec 1, 2010 at 5:09
  • I thought I could yesterday. But I can't. It seems that for a more specific search the number of products is about the same, although sorted differently.
    – Pullie
    Dec 1, 2010 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.