We recently switched e-commerce platform and due to poor planning, webmaster tools is reporting that thousands of URLs are 404. We obviously have to 301 redirect these and tell webmaster tools they have been fixed but how can we map the old URLs to their new counterparts without manually checking every single of the old ones using wayback machine (the old site is no longer on our server and the URL sets have completely different formats)?
-
Huge oversite. Gonna hurt you big time. That's a shame.– John Conde ♦Apr 17, 2014 at 23:38
-
I echo John's sentiments. There usually is a pattern, directory structure, or something that can help to make it a bit easier. Unfortunately, there is not enough info to answer your question. I recently changed aliases for 300,000 pages and I wrote code to create the redirects. It was not technically difficult, but still a bit of an effort. Without historical information, I cannot imagine how you are going to redirect thousands of pages without a lot of work and some late nights. Good luck! I wish I had an answer for you. I can, however, make a pot of coffee.– closetnocApr 18, 2014 at 0:00
3 Answers
I've worked on sites like this before where large sites were moved without redirects by people not understanding how redirects work. I had to use context clues found in the sites linking to the old version of the website to redirect the old URLs to the current ones. In one case the URLs weren't even friendly and that became a difficult task.
I divide the old links in to two categories:
- Old links that were frequently linked from other websites to the old pages
- Old links that didn't appear to have any incoming links from other websites.
Google is surprisingly forgiving of #2 and continues to show the new pages in equitable positions considering their lack of incoming links (most of their ranking comes from internal linking and domain authority). The pages falling into the #1 category however tend to get heavily affected because they are essentially new pages without a history. Your focus should be on repairing #1.
I make mapping tables in SQL to keep track of the old URLs and where they should point, then turn them into 301 redirects. Prioritize the URLs for best results.
You might still be able to get a decent amount of your old URLs from 3rd party SEO tools, something like Majestic, perhaps. Output their data to Excel, put the output of a crawl of your current site in another sheet, then use VLOOKUP
or similar to clean current URLs out of the 3rd party data.
If you need to resort to Wayback Machine, it is possible to write a script that will scrape URLs for a given domain en masse.
Mapping redirects
Typically, most of your traffic and revenue will come from a small number of your pages, so there's rarely a need to do 1:1 redirect mapping for all of your content anyway.
If you still have pre-migration data from your analytics package, you should be able to find those pages quite easily – sort pages by, e.g., traffic, conversions, etc. and you'll probably see a point where the numbers drop sharply.
Take those top-performing URLs and do your best to map them to their new equivalents. When URLs have changed completely, there's often no option but to do this manually, though if you have page titles that can make things easier. This is why I recommend the analytics bit: it makes sure you're expending the greatest effort on the most valuable content.
For everything else, hopefully you have enough data to match at category and sub-category level, e.g.:
/08867439_pink_widgets/.*
301 to /pink-widgets/
For things you can't match, you could just redirect to the homepage as a last resort. Some people don't like doing that, but in this sort of situation it's worth considering.
Whatever you do, be sure to optimise your 404 page: make sure it's branded and as helpful as possible in (a) encouraging visitors to stay on the site and (b) getting them to what they were looking for.
I've been in this situation before and I've built a 301 mapping tool to solve this problem. It's called GetRedirects
The tool is fully automated so you don't need to do manual mapping.
Just upload your list of 404s and it will create 1:1 redirects to the most relevant existing pages (it uses Natural Language Processing and Wayback Machine data for this).
Early access is free so you are welcome to give it a try!