Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We are currently trying to see if our site is being indexed successfully by google. Mind you we got millions of pages.

A trick some people use is the google site: command. I always assumed that google.com and google.nl would be the same. Data being fetched by the same GoogleBot. And that it was more of a latency thing where google.nl would be closer to home than google.com. But this is clearly not the case, since google.nl and google.com give back different results.

http://www.google.nl/search?sourceid=chrome&ie=UTF-8&q=site:amazon.com returns 37.400.000 results and http://www.google.com/#sclient=psy&hl=en&site=&source=hp&q=site:amazon.com returns 487,000,000 results

Does anybody know how this works?

Another related question is what other tricks people use to see if google.com successfully indexes all data.

It seems the best way is to do some analytics on the apache logs searching for the GoogleBot.

share|improve this question
    
I have always thought that the only the order is different but the results stays the same between different google domains. That is really, really strange. –  Ilian Iliev Nov 30 '10 at 11:30
    
"Local" Google sites have always given different results for me. I have used and/or tried at least 4 different ones. Each local site is optimized (or so it's supposed to be) for users of that particular domain. google.co.uk for instance gives results that are more relevant for visitors from the UK. –  Muhammad Mussnoon Nov 30 '10 at 13:24
    
Muhammed: But I don't see the optimization here. I took Amazon as an example, but a Dutch site slows less results on google.nl (dutch). I can't see this being a optimization. –  Pullie Nov 30 '10 at 21:12
    
I saw once (long ago) that the Google data centres are not always sync'd, so this is a reason why there are differences between the results on different sites. However, i think the example you show here goes far beyond differences in the data stored in various data centres. There's clearly some other logic going on. –  anthonyg Dec 1 '10 at 1:22
add comment

1 Answer

up vote 1 down vote accepted

Google Webmaster Tools is the easiest (and most authoritative) way to determine how much of your content has made it into Google's index.

share|improve this answer
    
Google Webmaster tools does not show how many pages are being indexed. It will say how many URL's are in the index based on the sitemap. Which in our case says 0. Something we try to discuss with Google since that is clearly not the case. And it does not explain the location difference. –  Pullie Nov 30 '10 at 12:20
    
@Ronald Pulleman - Valid points, however, the alternative to taking what Google freely provides isn't pretty. Last I checked a few years back, programs like Web CEO had difficulty with request throttling when one tried to grab search results en masse. Note that the numbers which Google returns as "total" for hits on a term are useless by Google's own admission (estimate differences by order of magnitude are not uncommon): google.com/support/webmasters/bin/… –  danlefree Nov 30 '10 at 15:29
    
danlefree: fair enough. However I also cannot find certain products in google.nl which I can find in google.com so there seems to be a difference in the index –  Pullie Nov 30 '10 at 21:38
    
@Ronald Pulleman - Can you give an example? (preferably something like "blue widget" site:yourdomain.com) –  danlefree Dec 1 '10 at 5:09
    
I thought I could yesterday. But I can't. It seems that for a more specific search the number of products is about the same, although sorted differently. –  Pullie Dec 1 '10 at 11:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.