New Magento version of a site about to launch. What the best way to create a list of URLs on my site that ensures I haven't missed any - so I can do redirects on them?
nd do I want to redirect all links that are indexed, or is there a reason not to do some? Thanks a lot for any guidance. For some reason the eComm/site admin guy here doesn't know the answer to this question.
For our conversion, I ran Xenu's Link Sleuth against the site to gather all the URLs on the site and exported it to a file I then imported into MS Access for processing.
I then purchased Scott Belosavich's URL Rewrite Import/Export module, did the full export to find the fields necessary for import, aligned this with the table I had and then generated all the redirect to links, set them all to be permanent 301 redirects per the field setting for that.
The import into Magento's core_url_rewrite index was uneventful and the 301s immediate for Google. For product pages that did not directly match, I set redirects to the nearest category so people had a general idea where to start.
Aside from the usual cringing white knuckle wait with the anticipated drop in traffic, between the sitemap and the redirects, Google was pretty quick in getting things changed over.
You also might use your Webmaster Tools account to mine ancient incoming links to see where product still exists despite the 5 year old 404 Google's been throwing and re-link to the appropriate new pages. We recovered about 20 rather good backlinks this way.
Get a month's worth of log files from your site. List all the URLs in your log file that have a 200 status. Here is a command line that will do so for my site:
cat access_log | grep '" 200 ' | sed 's/.*GET //g;s/ .*//g;' | sort | uniq
You could also run a web crawler against your site and have it list all your urls. I use wget recursively against my site. If you are on Windows, I have heard good things about Xenu Link Sleuth.
