For documents that get generated with a new version id, how should I handle the old URLs? Robots.txt, 301 redirect, or create permanent URLs?

Question

The site I am working on has around 800 products, some of these can have up to 2000 PDF and CAD drawings each.

The links to these PDF and CAD files are combined into a single object and then linked of a product page with the url structure of:

$domain/downloads/download/product/$combination_id

As there are so many drawings we have created a system that will completely delete the combinations and re-generate them (making new combination_id's in the process)

I have asked Google to not index these URLs and they don't appear in organic search results.

However Google Webmaster Tools has recently indexed the site and has thrown up a huge number of 404s for the old combination ids that have been deleted.

My question is, should I:

1. Block these urls with a /downloads/download/product robots.txt rule?

2. Create 301 redirects for each old combination that would then have to be updated whenever a new version of a drawing is added?

3. "Slugify" the combination_id's so they don't change when the drawings are updated and create 301 redirects for all the existing combinations.

Answer 1

By the sounds of it, this is really just one (of many) page that is being updated (with PDFs and CAD files), rather than an entirely new page - although that would seem to be how you have currently implemented this.

Consequently I think the URL should remain consistent and not change. So, this relates to your option #3: "Slugify the combination_id's so they don't change". You will need to implement 301 redirects for all "old" URLs (combination IDs).

If you blocked the old URLs with robots.txt then the URLs themselves could still remained indexed for an extended period of time, as the redirect will not be seen. Unless of course your intention is to block all product URLs (but I don't think you are really suggesting that)?