Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We're in the process of migrating an outdated e-commerce cart to Magento. Part of this process will entail configuring proper 301 redirects. I'm aiming to get a list of all indexed pages to throw our solution against and resolve as many potential issues as possible before going live.

Ideally I'd just need a CSV containing the URIs indexed by the Search Engines for our domain.

Looking at a similar question here, it seems there is no simple way to export this data from either Google or Bing's Webmaster tools, given that this cart has tens of thousands of products (and thus tens of thousands of indexed pages).

I've run across a few other 3rd party utilities such as Screaming Frog and web-based ones like searchenginegenie and internetmarketingninjas, but I've never used any of them and am hesitant to start throwing additional traffic at our site unless I know we'll get what we need out of it.

Has anyone out there used these tools to do something similar, or found some way to retrieve more than the top 1000 records from GWT (or something similar from Bing)?

share|improve this question
add comment

4 Answers 4

up vote 2 down vote accepted

I have used Screaming Frog, today in fact, and I really love that tool. You get a lot of information in quite a short amount of time. You'll get the metadata in a csv and you can manipulate it easily in Excel. Export everything and then use filters for each column to only display text/html and not images or CSS files.

I am doing that for a site migration right now and used it for one in the past. How many pages are you talking? Here's a comparison on Moz of Xenu vs SF.

share|improve this answer
1  
To add to this, Screaming Frog or Xenu won't specify which URL's from your website are indexed in search engines though. Great tools though all the same for crawling your website and outputting a list of URL's. –  zigojacko Jul 12 '13 at 7:53
1  
Two posts, two endorsements of Screaming Frog and a link to moz.com. Do you have any relationship with that product and site? –  John Conde Jul 12 '13 at 12:34
1  
GWT reports ~80k indexed pages. While getting the exact list of what's indexed on Google/etc would be nice, it's my understanding that it's not possible for more than 1k pages. I'll take a closer look at SF, thanks! –  JR.XYZA Jul 12 '13 at 20:03
    
I've spent some time with the trial version of Screaming Frog and it looks like the full version will do what we need quite nicely. Thanks for the recommendation and the helpful link! –  JR.XYZA Jul 16 '13 at 19:10
add comment

Whatever outdated e-commerce cart you are using now, if you are able to programatically generate all the product and category URLs in the old platform, you don't need to use 301 redirect. You can use the same URLs in Magento (by updating the core_url_rewrite table). This is a special feature of Magento.

I used to do SEO for a Magento website, which earlier used an outdated e-commerce cart just like yours. The migration was done to Magento keeping the same old URLs.

A word of caution about Magento. There is too much hype about Magento. Although it uses MVC architecture, it is one of the worst e-commerce platforms. This is especially true for large catalogs (true in your case, as you say you have tens of thousands of products and categories). Unless you are using the Enterprise Edition, which uses advanced caching to speed up things (and is very expensive), the Community edition will not help your cause.

They say Magento is SEO friendly. Nothing can be farther from the truth. Magento's own URLs (related to products and product reviews) are a complete mess as regards SEO. It generates multiple URLs (paths) for the same product, if it is assigned to multiple categories. In that case, you need to use Catalog Url Rewrite management anyway, which can be a headache.

Magneto's URL problems were solely responsible for causing a lot of damage to the business of the website I am talking about by hurting their search engine rankings.

Anyway, you shouldn't be bothering too much about which pages are in Google's index (which is multi-layered). Assume that all of them are indexed and 301 redirect all of them, if you decide to use 301 redirect.

Thanks,
Satyabrata Das

share|improve this answer
1  
Magento is easily made SEO friendly in the settings. You don't need to use redirects to fix multiple product URLs, just set the URL paths not to use category URLs, or if you do want the category URLs included in the URL, just use the canonical settings. –  Max Jul 12 '13 at 9:01
    
Our particular problem is that the old platform has bare product/category IDs passed in the URL via GET. Obviously we want to use friendlier URLs in Magento, hence the 301s. Thanks for expressing your opinions on working with Magento from an SEO perspective; these are all issues we're aware of and have plans for. –  JR.XYZA Jul 12 '13 at 20:08
add comment

You can download as .csv indexed webpages by Google in your Google Webmaster Tools account.

Just see Download chart data button below chart of indexed webpages (Menu Google Index => Indexation state).

share|improve this answer
1  
"Download chart data" does not download a list of indexed URLs, it simply downloads the chart data... a list of dates and numbers. –  w3d Jul 12 '13 at 10:33
add comment

You could also search for site:mydomain.com on Google to get a list of all their indexed pages from your domain, including subdomains.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.