We submit a sitemap to google and can see this being indexed in webmaster tools. We have 5140 entries, and have broken this up to 10 child site maps of 500 each. This all seems to be working well.

Google, however is not indexing all entries of three of the child site maps (all others seem to be nicely indexed). The number of entries indexed has remained static for the last 6 weeks.

We would now like to determine which url's are not being indexed by google, to try and see if there is a content issue or otherwise.

Is there any way to determine which url's are not being added to the index other than manually go through all 500 entries using 'site' on google?

  • You can use curl instead of doing it manually (be careful to use a timeout before performing another search, or Google will ban your IP for an hour or so), especially for large number of URLs. But I do not know any official Google way.
    – milo5b
    Jul 18, 2012 at 10:31
  • Sounds like an idea (and am aware of the google issues, having used a rank tracking tool). As someone who has not really used cURL before, do you have a rough guideline/direction on how to do this. Happy to google details, but any directional indication on which to use would be appreciated!
    – C4PO
    Jul 18, 2012 at 11:03

1 Answer 1


I will reply as an answer because I cant fit everything in a comment:

You could use a few tools, but I will describe the way I am most comfortable with, PHP + cURL + DOMDocument.

First you need to generate the search queries, so probably reading your sitemap.

$url = 'www.google.com/search?q=' . urlencode("site:".$sitemap_url);

(add more parameters to search url if needed)

Then you probably want to fake your useragent and set other cURL options if needed.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Your fake useragent string here');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$out = curl_exec($ch);

$dom = new DOMDocument();

Now that you have Google search loaded in a DOMDocument, you can parse it and look if your url is actually present in the search. If it is, it's indexed.

Hope this helps

  • 1
    You'll need to be careful with this - automated scripts fetching Google results can get banned. Jul 18, 2012 at 13:41
  • Yes, I have commented about that earlier in the question. You would need a sleep (with some randomness) every fetch.
    – milo5b
    Jul 18, 2012 at 14:17
  • Thanks, appreciate this. Will check to see if this is within google terms of service, first.
    – C4PO
    Jul 18, 2012 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.