Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We submit a sitemap to google and can see this being indexed in webmaster tools. We have 5140 entries, and have broken this up to 10 child site maps of 500 each. This all seems to be working well.

Google, however is not indexing all entries of three of the child site maps (all others seem to be nicely indexed). The number of entries indexed has remained static for the last 6 weeks.

We would now like to determine which url's are not being indexed by google, to try and see if there is a content issue or otherwise.

Is there any way to determine which url's are not being added to the index other than manually go through all 500 entries using 'site' on google?

share|improve this question
    
You can use curl instead of doing it manually (be careful to use a timeout before performing another search, or Google will ban your IP for an hour or so), especially for large number of URLs. But I do not know any official Google way. –  milo5b Jul 18 '12 at 10:31
    
Sounds like an idea (and am aware of the google issues, having used a rank tracking tool). As someone who has not really used cURL before, do you have a rough guideline/direction on how to do this. Happy to google details, but any directional indication on which to use would be appreciated! –  plantify Jul 18 '12 at 11:03
add comment

1 Answer 1

up vote 2 down vote accepted

I will reply as an answer because I cant fit everything in a comment:

You could use a few tools, but I will describe the way I am most comfortable with, PHP + cURL + DOMDocument.

First you need to generate the search queries, so probably reading your sitemap.

$url = 'www.google.com/search?q=' . urlencode("site:".$sitemap_url);

(add more parameters to search url if needed)

Then you probably want to fake your useragent and set other cURL options if needed.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Your fake useragent string here');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$out = curl_exec($ch);

$dom = new DOMDocument();
@$dom->loadHTML($output);

Now that you have Google search loaded in a DOMDocument, you can parse it and look if your url is actually present in the search. If it is, it's indexed.

Hope this helps

share|improve this answer
1  
You'll need to be careful with this - automated scripts fetching Google results can get banned. –  DisgruntledGoat Jul 18 '12 at 13:41
    
Yes, I have commented about that earlier in the question. You would need a sleep (with some randomness) every fetch. –  milo5b Jul 18 '12 at 14:17
    
Thanks, appreciate this. Will check to see if this is within google terms of service, first. –  plantify Jul 18 '12 at 15:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.