1

I run an eCommerce website and I have been thinking about adding PDF downloads to my product pages. These manuals and product sheets are typically hosted by the product manufacturer, though not all resources are safely linkable, since some URLs can often change. In order to avoid possible 404s, I was planning to write a simple link checker, a quick curl request to check if the result is still a 200 OK code.

If a non 200 code were detected, I could fall back to the locally hosted version, so I would still be providing my users with the information they need, while at the same time, I would only be hosting duplicate content if the original location was unavailable.

Now, is this a good idea? I suspect that hosting a file that exists elsewhere could do more harm than good. I am very interested in helping my SEO, rather than hurting it, so I am trying to plan every step I take in advance.

2

Any time that you host content that is published elsewhere you need to consider copyright law and duplicate content SEO.

To host files yourself, you would need permission from the manufacturers to copy the files. I assume that this wouldn't be a problem for most of them because they are trying to sell a product, not protect the contents of the manuals that go with it. However, some may not want old versions of their manuals floating around on other websites. If you can't get the permission, it wouldn't be wise to host the files yourself.

If you want to host the duplicated PDF files on your own site, there are ways you can manage it from an SEO standpoint.

The easiest way would be to not let Google crawl the duplicate content. Put them all in a directory (like /product-manuals/) and disallow crawling in robots.txt

User-agent: *
Disallow: /product-manuals/

Instead you could choose to configure your webserver with an HTTP header that would prevent Google from indexing those files:

X-Robots-Tag: noindex
  • I will be doing it the way you say. Thank you very much. – Juanpi Jan 12 '16 at 23:03
3

Your way of thinking is good for development and testing purposes, Not in production level as you are making external calls, every time users visit a product (Even if they don't actually want to get the file!) Which is horrible.

I would suggest hosting all PDF files on your own server (If it is not against the manufacturer copyright policy).

Two advantages are:

  1. Considerably faster file access for users.
  2. Prevents any broken link, 404 or even wrong files!
  3. Peace of mind (As a bonus advantage)!
  • I was actually thinking about running a periodic process (cron) to check the links. While it's still possible for a user to find a 404, a delay of, say, 4 hours isn't that bad either. What truly worries me is the SEO effect that hosting duplicated PDFs could have. – Juanpi Jan 12 '16 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.