How to map thousands of URLs for 301 redirection?

Question

We recently switched e-commerce platform and due to poor planning, webmaster tools is reporting that thousands of URLs are 404. We obviously have to 301 redirect these and tell webmaster tools they have been fixed but how can we map the old URLs to their new counterparts without manually checking every single of the old ones using wayback machine (the old site is no longer on our server and the URL sets have completely different formats)?

Answer 1

I've worked on sites like this before where large sites were moved without redirects by people not understanding how redirects work. I had to use context clues found in the sites linking to the old version of the website to redirect the old URLs to the current ones. In one case the URLs weren't even friendly and that became a difficult task.

I divide the old links in to two categories:

1. Old links that were frequently linked from other websites to the old pages
2. Old links that didn't appear to have any incoming links from other websites.

Google is surprisingly forgiving of #2 and continues to show the new pages in equitable positions considering their lack of incoming links (most of their ranking comes from internal linking and domain authority). The pages falling into the #1 category however tend to get heavily affected because they are essentially new pages without a history. Your focus should be on repairing #1.

I make mapping tables in SQL to keep track of the old URLs and where they should point, then turn them into 301 redirects. Prioritize the URLs for best results.

Answer 2

You might still be able to get a decent amount of your old URLs from 3rd party SEO tools, something like Majestic, perhaps. Output their data to Excel, put the output of a crawl of your current site in another sheet, then use VLOOKUP or similar to clean current URLs out of the 3rd party data.

If you need to resort to Wayback Machine, it is possible to write a script that will scrape URLs for a given domain en masse.

Mapping redirects

Typically, most of your traffic and revenue will come from a small number of your pages, so there's rarely a need to do 1:1 redirect mapping for all of your content anyway.

If you still have pre-migration data from your analytics package, you should be able to find those pages quite easily – sort pages by, e.g., traffic, conversions, etc. and you'll probably see a point where the numbers drop sharply.

Take those top-performing URLs and do your best to map them to their new equivalents. When URLs have changed completely, there's often no option but to do this manually, though if you have page titles that can make things easier. This is why I recommend the analytics bit: it makes sure you're expending the greatest effort on the most valuable content.

For everything else, hopefully you have enough data to match at category and sub-category level, e.g.:

/08867439_pink_widgets/.* 301 to /pink-widgets/

For things you can't match, you could just redirect to the homepage as a last resort. Some people don't like doing that, but in this sort of situation it's worth considering.

Whatever you do, be sure to optimise your 404 page: make sure it's branded and as helpful as possible in (a) encouraging visitors to stay on the site and (b) getting them to what they were looking for.

Answer 3

I've been in this situation before and I've built a 301 mapping tool to solve this problem. It's called GetRedirects

The tool is fully automated so you don't need to do manual mapping.

Just upload your list of 404s and it will create 1:1 redirects to the most relevant existing pages (it uses Natural Language Processing and Wayback Machine data for this).

Early access is free so you are welcome to give it a try!