I have the following three domains:

  1. example1.com (Disallow)
  2. example2.com (Disallow)
  3. example3.com (allow)

All domain points to same folder which is public_html.

How to disallow search engines from crawling pages on the first two domains?

  • 1
    How are you "pointing" all three domains to public_html?
    – zigojacko
    Aug 12, 2015 at 7:03
  • 1
    What server software are you using?
    – MrWhite
    Aug 12, 2015 at 7:17

3 Answers 3


You need to conditionally serve a different robots.txt file based on which domain/host has been accessed. On Apache you can do this in .htaccess:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(example1\.com|example2\.com)$
RewriteRule ^robots\.txt$ /robots-disallow.txt [L]

This is specific to your example, where it will serve a "disallow" robots.txt (robots-disallow.txt) to #1 and #2, otherwise the request will fall through and serve your regular robots.txt file for #3 which "allows".


I work on a similar website where I manage this with PHP.

In my .htaccess I have a line reading:

RewriteRule ^robots.txt$ robots.php

In robots.php I have the following:

  header('Content-Type: text/plain; charset=utf-8');

    echo 'User-agent: *
    Host: https://www.allowed-site.fr/
    Sitemap: https://www.allowed-site.fr/sitemap-fr.xml
  } else if($_SERVER['HTTP_HOST']=='www.allowed-site.lu'){
    echo 'User-agent: *
    Disallow: /example/
    Host: https://www.allowed-site.lu/
    Sitemap: https://www.allowed-site.lu/sitemap-lu.xml
  } else {
    echo 'User-agent: *
    Disallow: /

This allows you to have a single file where you can tailor your robots.txt for each individual domain.


Since you're allowing search engines to only one domain, your RewriteRules can be made simpler. Just use this:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^example3\.com$
RewriteRule ^robots\.txt$ /robots-allow.txt [L]

In the public_html folder you need to create two files. robots.txt and robots-allow.txt

In robots.txt you need to add the following:

User-agent: searchengine
Disallow: /

replace searchengine with google or whatever engine you want to block. Repeat the above lines if you want to add more search engines. Then your robots.txt will turn out to be something like this:

User-agent: searchengine
Disallow: /
User-agent: searchengine2
Disallow: /
User-agent: searchengine3
Disallow: /

Then in robots-allow.txt you could leave it as a blank file or adjust the crawl delay for all search engines like this:

User-agent: *
Crawl-delay: 1

The number after crawl-delay represents the minimum waiting time in seconds between requests to the server from the same user agent.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.