Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

New Magento version of a site about to launch. What the best way to create a list of URLs on my site that ensures I haven't missed any - so I can do redirects on them?

nd do I want to redirect all links that are indexed, or is there a reason not to do some? Thanks a lot for any guidance. For some reason the eComm/site admin guy here doesn't know the answer to this question.

share|improve this question
    
I would crawl the site :) –  milo5b Oct 11 '12 at 18:49
1  
You don't say whether they are dynamic URLs or not. But try using Xenu home.snafu.de/tilman/xenulink.html –  Steve Oct 12 '12 at 0:16
    
What is a "Magento version"? –  Arjang Oct 13 '12 at 9:05
    
eCommerce platform based on Zend Framework. –  Fiasco Labs Apr 24 '13 at 1:24
add comment

2 Answers 2

For our conversion, I ran Xenu's Link Sleuth against the site to gather all the URLs on the site and exported it to a file I then imported into MS Access for processing.

I then purchased Scott Belosavich's URL Rewrite Import/Export module, did the full export to find the fields necessary for import, aligned this with the table I had and then generated all the redirect to links, set them all to be permanent 301 redirects per the field setting for that.

The import into Magento's core_url_rewrite index was uneventful and the 301s immediate for Google. For product pages that did not directly match, I set redirects to the nearest category so people had a general idea where to start.

Aside from the usual cringing white knuckle wait with the anticipated drop in traffic, between the sitemap and the redirects, Google was pretty quick in getting things changed over.

You also might use your Webmaster Tools account to mine ancient incoming links to see where product still exists despite the 5 year old 404 Google's been throwing and re-link to the appropriate new pages. We recovered about 20 rather good backlinks this way.

share|improve this answer
add comment

Get a month's worth of log files from your site. List all the URLs in your log file that have a 200 status. Here is a command line that will do so for my site:

cat access_log |  grep '" 200 ' | sed 's/.*GET //g;s/ .*//g;' | sort | uniq

You could also run a web crawler against your site and have it list all your urls. I use wget recursively against my site. If you are on Windows, I have heard good things about Xenu Link Sleuth.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.