Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have the following three domains:

  1. example1.com (Disallow)
  2. example2.com (Disallow)
  3. example3.com (allow)

All domain points to same folder which is public_html.

How to disallow search engines from crawling pages on the first two domains?

share|improve this question
1  
How are you "pointing" all three domains to public_html? –  zigojacko Aug 12 at 7:03
1  
What server software are you using? –  w3d Aug 12 at 7:17

3 Answers 3

You need to conditionally serve a different robots.txt file based on which domain/host has been accessed. On Apache you can do this in .htaccess:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(example1\.com|example2\.com)$
RewriteRule ^robots\.txt$ /robots-disallow.txt [L]

This is specific to your example, where it will serve a "disallow" robots.txt (robots-disallow.txt) to #1 and #2, otherwise the request will fall through and serve your regular robots.txt file for #3 which "allows".

share|improve this answer

I work on a similar website where I manage this with PHP.

In my .htaccess I have a line reading:

RewriteRule ^robots.txt$ robots.php

In robots.php I have the following:

<?php
  header('Content-Type: text/plain; charset=utf-8');

  if($_SERVER['HTTP_HOST']=='www.allowed-site.fr'){
    echo 'User-agent: *
    Disallow: 
    Host: https://www.allowed-site.fr/
    Sitemap: https://www.allowed-site.fr/sitemap-fr.xml
    ';
  } else if($_SERVER['HTTP_HOST']=='www.allowed-site.lu'){
    echo 'User-agent: *
    Disallow: /example/
    Host: https://www.allowed-site.lu/
    Sitemap: https://www.allowed-site.lu/sitemap-lu.xml
    ';
  } else {
    echo 'User-agent: *
    Disallow: /
    ';
  }
?>

This allows you to have a single file where you can tailor your robots.txt for each individual domain.

share|improve this answer

Since you're allowing search engines to only one domain, your RewriteRules can be made simpler. Just use this:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^example3\.com$
RewriteRule ^robots\.txt$ /robots-allow.txt [L]

In the public_html folder you need to create two files. robots.txt and robots-allow.txt

In robots.txt you need to add the following:

User-agent: searchengine
Disallow: /

replace searchengine with google or whatever engine you want to block. Repeat the above lines if you want to add more search engines. Then your robots.txt will turn out to be something like this:

User-agent: searchengine
Disallow: /
User-agent: searchengine2
Disallow: /
User-agent: searchengine3
Disallow: /

Then in robots-allow.txt you could leave it as a blank file or adjust the crawl delay for all search engines like this:

User-agent: *
Crawl-delay: 1

The number after crawl-delay represents the minimum waiting time in seconds between requests to the server from the same user agent.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.