I have the following three domains:
All domain points to same folder which is
How to disallow search engines from crawling pages on the first two domains?
You need to conditionally serve a different robots.txt file based on which domain/host has been accessed. On Apache you can do this in .htaccess:
This is specific to your example, where it will serve a "disallow" robots.txt (
I work on a similar website where I manage this with PHP.
In my .htaccess I have a line reading:
In robots.php I have the following:
This allows you to have a single file where you can tailor your robots.txt for each individual domain.
Since you're allowing search engines to only one domain, your RewriteRules can be made simpler. Just use this:
In the public_html folder you need to create two files. robots.txt and robots-allow.txt
In robots.txt you need to add the following:
replace searchengine with google or whatever engine you want to block. Repeat the above lines if you want to add more search engines. Then your robots.txt will turn out to be something like this:
Then in robots-allow.txt you could leave it as a blank file or adjust the crawl delay for all search engines like this:
The number after crawl-delay represents the minimum waiting time in seconds between requests to the server from the same user agent.