I have a website with about 1.5M pages. I am condensing pages that do not have large amounts of content to improve the quality of the site. As a result, I have about 70K pages that I have added a canonical link to, pointing towards the new master page for the content.

Now I want to completely work the existing pages out of the system. I want to 301 redirect the existing pages to the new master pages that the content now resides on. I am running IIS and normally would use RewriteMaps (Apache). I am concerned that adding 70k lines to my web.config, to be parsed for every URL, is going to affect performance.

Any suggestions on a known/tested way to accomplish this?

  • 2
    What powers the pages? Are they static pages or is they dynamically generated? If dynamic, what programming language? Commented Sep 15, 2013 at 23:57
  • 1
    static..... If they were dynamic it wouldn't be a problem. Commented Sep 16, 2013 at 1:33
  • 2
    Is there a pattern to them such that they could be implemented with small number of rules based on regular expressions? Commented Sep 16, 2013 at 9:37
  • Find out the pattern to make is simpler in few codes.
    – user2434
    Commented Sep 16, 2013 at 11:15
  • There are no solid patterns. The files reside in many different folders and are named as the first sentence of the document. TY for the help, I think I may just have to insert a < meta> redirect with no refresh time in each file. Commented Sep 16, 2013 at 18:16

1 Answer 1


There are two solutions I can think of:

  1. Find patterns in the redirects and see if you can cut the number down. For example, if you had a large chunk that redirected from one folder to another, you could match something like /folder/([A-Za-z0-9-]+) and redirect to /newfolder/$1. This might still leave a large number of redirects though.
  2. Redirect all unfound pages to one script, where you look up a list of redirects in a database or other file. If you find a match then redirect, otherwise serve a 404. Using only one redirect at the server level means serving all your regular pages lose no speed. Your redirects may technically be slower but it should likely be negligible, and they will not be the most popular pages.
  • I think I can do something with number 2 with a custom 404 page. By only performing the long look up on pages not found versus every page will not affect my good traffic at all. Commented Sep 23, 2013 at 14:32
  • I am extremely happy with the custom 404 solution, it works perfect for what I am doing. TY for the help. Commented Sep 30, 2013 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.