Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am looking for a way to take every link that I have indexed by Google and export them to a CSV file. Recently I have had way more pages indexed by Google then I actually have and I want to find where all these pages are coming from without having to view each search result page.

share|improve this question
    
Where have you gotten the number of indexed pages from? –  w3d Aug 28 '12 at 17:45
    
Google Webmaster and searching site:domain.com –  Lee Aug 28 '12 at 18:04
2  
The only thing I would say is that the data reported in Webmaster Tools (Health > Index Status > Total Indexed) will be more accurate than that reported by a site:domain.com search. A site: search always returns a much higher figure in my experience, but if you step through the SERPs, the actual number of results is less than the "About NNNN results" figure. –  w3d Aug 28 '12 at 18:38
    
Well that page was the reason why I got interested in this in the first place. In 3 months, the number of indexed pages has gone from 27,000 to 567,000 and I want to know why. –  Lee Aug 28 '12 at 18:56
add comment

3 Answers

up vote 5 down vote accepted

Unfortunately there is no way to get a full list of every indexed page in Google. Even milo5b's solution will only get you at most 1,000 URLs.

It sounds like you have some duplicate content issues. In Webmaster Tools, check in Health > Index Status and it will show you a cumulative total of pages indexed over time. If the graph makes a big leap at one point you may be able to work out if a specific change on your site triggered the jump.

You could also try using Bing's Webmaster Tools. They have an Index Explorer which could help you find the URLs. Search engine spiders are quite similar so if Google found those links, Bing probably did too.

I thought Bing had a way to export most of its data but I cannot find it on a cursory glance. There is an API though so you could probably use that to extract everything.

share|improve this answer
    
Thank you for the Bing suggestion but they only had 9,000 pages indexed and I am fairly sure they are not the links I needed. –  Lee Aug 29 '12 at 16:17
    
This is an awesome tip, thanks for sharing. –  user23401 Feb 15 '13 at 15:23
add comment

I ended up drilling down to the problematic sub-folder through searching for site:domain.com/foo/bar/ but in my search I did come across a method for getting the search results into an excel file.

Open up a Google Docs spreadsheet and use this formula:

=importXml("www.google.com/search?q=site:domain.com&num=100&start=1"; "//cite")

It will only get the first 100 results but you can use it again to get the next 100. Just change the start variable:

=importXml("www.google.com/search?q=site:domain.com&num=100&start=100"; "//cite")

This will only provide up to 1000 results, as mentioned previously by DisgruntledGoat, but the formula can be changed to provide links from specific sub-directories:

=importXml("www.google.com/search?q=site:domain.com/foo/bar/&num=100&start=1"; "//cite")

share|improve this answer
    
Great tip with Google Docs. Just wondering what the actual problem was with regard to the extra indexed pages - was it duplicate content? –  w3d Aug 30 '12 at 11:36
1  
I traced it back to vBulletin, a forum software we are using. They added a new feature called activity stream and added into the users section. So every user would not only have pages of their own activity on their profile but all the activity of every friend they have. On top of the Google was indexing blank activity pages because vBulletin would not return a 404. I ended up no-indexing the whole section. –  Lee Aug 30 '12 at 15:46
add comment

You could write a script that parses Google's SERP (for example PHP + Curl) and store each link in a CSV file. Be careful to have your script behave like a human, because Google could ban your IP from search results for a few hours if you abuse this.

share|improve this answer
add comment

protected by Community Mar 7 '13 at 10:17

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.