We recently switched e-commerce platform and due to poor planning, webmaster tools is reporting that thousands of URLs are 404. We obviously have to 301 redirect these and tell webmaster tools they have been fixed but how can we map the old URLs to their new counterparts without manually checking every single of the old ones using wayback machine (the old site is no longer on our server and the URL sets have completely different formats)?
I've worked on sites like this before where large sites were moved without redirects by people not understanding how redirects work. I had to use context clues found in the sites linking to the old version of the website to redirect the old URLs to the current ones. In one case the URLs weren't even friendly and that became a difficult task.
I divide the old links in to two categories:
Google is surprisingly forgiving of #2 and continues to show the new pages in equitable positions considering their lack of incoming links (most of their ranking comes from internal linking and domain authority). The pages falling into the #1 category however tend to get heavily affected because they are essentially new pages without a history. Your focus should be on repairing #1.
I make mapping tables in SQL to keep track of the old URLs and where they should point, then turn them into 301 redirects. Prioritize the URLs for best results.
You might still be able to get a decent amount of your old URLs from 3rd party SEO tools, something like Majestic, perhaps. Output their data to Excel, put the output of a crawl of your current site in another sheet, then use
If you need to resort to Wayback Machine, it is possible to write a script that will scrape URLs for a given domain en masse.
Typically, most of your traffic and revenue will come from a small number of your pages, so there's rarely a need to do 1:1 redirect mapping for all of your content anyway.
If you still have pre-migration data from your analytics package, you should be able to find those pages quite easily – sort pages by, e.g., traffic, conversions, etc. and you'll probably see a point where the numbers drop sharply.
Take those top-performing URLs and do your best to map them to their new equivalents. When URLs have changed completely, there's often no option but to do this manually, though if you have page titles that can make things easier. This is why I recommend the analytics bit: it makes sure you're expending the greatest effort on the most valuable content.
For everything else, hopefully you have enough data to match at category and sub-category level, e.g.:
For things you can't match, you could just redirect to the homepage as a last resort. Some people don't like doing that, but in this sort of situation it's worth considering.
Whatever you do, be sure to optimise your 404 page: make sure it's branded and as helpful as possible in (a) encouraging visitors to stay on the site and (b) getting them to what they were looking for.