4

I am a web developer and I want to create clones of the sites I've developed for clients, so that I have an "original copy" on a subdomain of my own website, so that I can showcase my work to new clients.

What is the best way to not get my clients original websites penalised for duplicate content?

I am planning to have a robots.txt file that disallows all robots, as well as using

<link href="http://www.client-canonical-site.com/" rel="canonical" />

in the <head> of the pages.

Is that sufficient? Should I use rel=nofollow on all the links as well?

Improve this question
1
  • robots.txt is not enough to keep the crawlers from indexing the pages, see my answer
    – Anagio
    Commented Mar 20, 2012 at 15:38

4 Answers 4

Reset to default
8

Use robots.txt and put the following inside:

User-agent: *
Disallow: /

That's really all you need. Also, if a client's website is indexed as the original one, it won't get a penalty from duplicates.

Improve this answer
2

The problem with web bots is that you have to assume that they will follow all the rules that you've set out.

If they're going to follow any of them, it will be the robots.txt file, so having just that that should be enough. However, the rest won't hurt.

Improve this answer
3
  • Thanks -- I'm assuming the major search engines, like Google, Bing, Yahoo, etc will follow the rules (since they more or less set them). I'm not concerned about what other bots do. I just don't want to penalise my clients website because I have duplicates of their content...
    – gillespieza
    Commented Dec 3, 2010 at 12:02
  • I think this answer sums it up well. The canonical is a good failsafe for if a bot ignores your robots.txt
    – Stephan Muller
    Commented Dec 3, 2010 at 12:37
  • @Amanda You can be confident that any robot from a respectable search engine will honor robots.txt. It is in their best interest to do so. And if they never crawl something - because its disallowed by robots.txt - they do not know it exists.
    – Kris
    Commented Dec 3, 2010 at 12:39
0

The best solution is rel="canonical". Some robots are bad robots and will crawl your page, then they will place links on their SERP's and after that Google will know about them. I've tested this with one of my websites, and some of the links was indexed, even there are rule - User-agent: * Disallow: /

Good luck

Improve this answer
0

robots.txt is not going to stop Google or others from Indexing the blocked pages. It's simply asking them to not crawl these folders.

If however your main domain, or a client links directly to the blocked pages Google will index the page.

Be sure to add noindex and nofollow to those pages meta tags, but to be safest you should password protect the folder they are in.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.