I have a website with about 1.5M pages. I am condensing pages that do not have large amounts of content to improve the quality of the site. As a result I have about 70K pages that I have added a < canonical link> to, pointing towards the new master page for the content. Now I want to completely work the existing pages out of the system. I want to 301 redirect the existing pages to the new master pages that the content now resides upon. I am running IIS and normally would use < rewriteMaps>. I am concerned that adding 70k lines to my web.config, to be parsed for every URL, is going to affect performance. Any suggestions on a known/tested way to accomplish this? TY for your time.

  • 2
    What powers the pages? Are they static pages or is they dynamically generated? If dynamic, what programming language? – Stephen Ostermiller Sep 15 '13 at 23:57
  • 1
    static..... If they were dynamic it wouldn't be a problem. – Vincent James Sep 16 '13 at 1:33
  • 2
    Is there a pattern to them such that they could be implemented with small number of rules based on regular expressions? – Stephen Ostermiller Sep 16 '13 at 9:37
  • Find out the pattern to make is simpler in few codes. – user2434 Sep 16 '13 at 11:15
  • There are no solid patterns. The files reside in many different folders and are named as the first sentence of the document. TY for the help, I think I may just have to insert a < meta> redirect with no refresh time in each file. – Vincent James Sep 16 '13 at 18:16

There are two solutions I can think of:

  1. Find patterns in the redirects and see if you can cut the number down. For example, if you had a large chunk that redirected from one folder to another, you could match something like /folder/([A-Za-z0-9-]+) and redirect to /newfolder/$1. This might still leave a large number of redirects though.
  2. Redirect all unfound pages to one script, where you look up a list of redirects in a database or other file. If you find a match then redirect, otherwise serve a 404. Using only one redirect at the server level means serving all your regular pages lose no speed. Your redirects may technically be slower but it should likely be negligible, and they will not be the most popular pages.
| improve this answer | |
  • I think I can do something with number 2 with a custom 404 page. By only performing the long look up on pages not found versus every page will not affect my good traffic at all. – Vincent James Sep 23 '13 at 14:32
  • I am extremely happy with the custom 404 solution, it works perfect for what I am doing. TY for the help. – Vincent James Sep 30 '13 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.