I am looking for a way to take every link that I have indexed by Google and export them to a CSV file. Recently I have had way more pages indexed by Google then I actually have and I want to find where all these pages are coming from without having to view each search result page.

  • Where have you gotten the number of indexed pages from?
    – MrWhite
    Commented Aug 28, 2012 at 17:45
  • Google Webmaster and searching site:domain.com
    – Lee
    Commented Aug 28, 2012 at 18:04
  • 2
    The only thing I would say is that the data reported in Webmaster Tools (Health > Index Status > Total Indexed) will be more accurate than that reported by a site:domain.com search. A site: search always returns a much higher figure in my experience, but if you step through the SERPs, the actual number of results is less than the "About NNNN results" figure.
    – MrWhite
    Commented Aug 28, 2012 at 18:38
  • Well that page was the reason why I got interested in this in the first place. In 3 months, the number of indexed pages has gone from 27,000 to 567,000 and I want to know why.
    – Lee
    Commented Aug 28, 2012 at 18:56

3 Answers 3


I ended up drilling down to the problematic sub-folder through searching for site:domain.com/foo/bar/ but in my search I did come across a method for getting the search results into an excel file.

Open up a Google Docs spreadsheet and use this formula:

=importXml("www.google.com/search?q=site:domain.com&num=100&start=1"; "//cite")

It will only get the first 100 results but you can use it again to get the next 100. Just change the start variable:

=importXml("www.google.com/search?q=site:domain.com&num=100&start=100"; "//cite")

This will only provide up to 1000 results, as mentioned previously by DisgruntledGoat, but the formula can be changed to provide links from specific sub-directories:

=importXml("www.google.com/search?q=site:domain.com/foo/bar/&num=100&start=1"; "//cite")

  • Great tip with Google Docs. Just wondering what the actual problem was with regard to the extra indexed pages - was it duplicate content?
    – MrWhite
    Commented Aug 30, 2012 at 11:36
  • 1
    I traced it back to vBulletin, a forum software we are using. They added a new feature called activity stream and added into the users section. So every user would not only have pages of their own activity on their profile but all the activity of every friend they have. On top of the Google was indexing blank activity pages because vBulletin would not return a 404. I ended up no-indexing the whole section.
    – Lee
    Commented Aug 30, 2012 at 15:46
  • importXML only works properly with the old Sheets which can be activated with this link: g.co/oldsheets
    – i.amniels
    Commented May 22, 2015 at 11:29

Unfortunately there is no way to get a full list of every indexed page in Google. Even milo5b's solution will only get you at most 1,000 URLs.

It sounds like you have some duplicate content issues. In Webmaster Tools, check in Health > Index Status and it will show you a cumulative total of pages indexed over time. If the graph makes a big leap at one point you may be able to work out if a specific change on your site triggered the jump.

You could also try using Bing's Webmaster Tools. They have an Index Explorer which could help you find the URLs. Search engine spiders are quite similar so if Google found those links, Bing probably did too.

I thought Bing had a way to export most of its data but I cannot find it on a cursory glance. There is an API though so you could probably use that to extract everything.

  • Thank you for the Bing suggestion but they only had 9,000 pages indexed and I am fairly sure they are not the links I needed.
    – Lee
    Commented Aug 29, 2012 at 16:17

You could write a script that parses Google's SERP (for example PHP + Curl) and store each link in a CSV file. Be careful to have your script behave like a human, because Google could ban your IP from search results for a few hours if you abuse this.

Not the answer you're looking for? Browse other questions tagged or ask your own question.