Duplicate content: Prevent testing and development site from getting crawled and indexed

Question

I am running a WordPress based site on example.com with its original content.

I then have another site running on example.me which is basically the exact copy of the original example.com together with all example.com's content and I'm using this example.me only for testing purposes, to develop and test new plugins and functionality which I then deploy on the main domain example.com.

I use the default WordPress settings on example.me that disables robots from crawling the site, but WordPress warns that this may not always work.

Is there any other way to let Google know straight away that this site example.me is a duplicate of example.com and is not be indexed?

Some kind of general canonical tag that I could use for the entire site?

Is there maybe a setting in webmaster tools to have this done?

Answer 1

Canonical tags are not designed to work domain wide, they are designed to identify the canonical link on a page by page basis. The best way is to use a robots.txt file containing the following...

User-agent: *
Disallow: /

What this robots.txt file is telling the crawlers is not to crawl or index the site at all. Now whether or not a crawler respects this file is up to them and so can't be used to prevent robots access for security reasons however for the purpose you've mentioned given the site is full of duplicate content and is designed as a development and testing environment it will work and prevent the site from being indexed and crawled by Google for indexing reasons.