Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am a web developer and I want to create clones of the sites I've developed for clients, so that I have an "original copy" on a subdomain of my own website, so that I can showcase my work to new clients.

What is the best way to not get my clients original websites penalised for duplicate content?

I am planning to have a robots.txt file that disallows all robots, as well as using

<link href="http://www.client-canonical-site.com/" rel="canonical" />

in the <head> of the pages.

Is that sufficient? Should I use rel=nofollow on all the links as well?

share|improve this question
robots.txt is not enough to keep the crawlers from indexing the pages, see my answer – Anagio Mar 20 '12 at 15:38
up vote 6 down vote accepted

Use robots.txt and put the following inside:

User-agent: *
Disallow: /

That's really all you need. Also, if a client's website is indexed as the original one, it won't get a penalty from duplicates.

share|improve this answer

The problem with web bots is that you have to assume that they will follow all the rules that you've set out.

If they're going to follow any of them, it will be the robots.txt file, so having just that that should be enough. However, the rest won't hurt.

share|improve this answer
Thanks -- I'm assuming the major search engines, like Google, Bing, Yahoo, etc will follow the rules (since they more or less set them). I'm not concerned about what other bots do. I just don't want to penalise my clients website because I have duplicates of their content... – Amanda Dec 3 '10 at 12:02
I think this answer sums it up well. The canonical is a good failsafe for if a bot ignores your robots.txt – Stephan Muller Dec 3 '10 at 12:37
@Amanda You can be confident that any robot from a respectable search engine will honor robots.txt. It is in their best interest to do so. And if they never crawl something - because its disallowed by robots.txt - they do not know it exists. – Kris Dec 3 '10 at 12:39

The best solution is rel="canonical". Some robots are bad robots and will crawl your page, then they will place links on their SERP's and after that Google will know about them. I've tested this with one of my websites, and some of the links was indexed, even there are rule - User-agent: * Disallow: /

Good luck

share|improve this answer

robots.txt is not going to stop Google or others from Indexing the blocked pages. It's simply asking them to not crawl these folders.

If however your main domain, or a client links directly to the blocked pages Google will index the page.

Be sure to add noindex and nofollow to those pages meta tags, but to be safest you should password protect the folder they are in.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.