1

I have a single web host with a number of other parked domains/sites in sub-directories, like this:

example.com is the primary site and root directory of the web hosting.

example.com/www.example2.com is one of the parked sites, but it is just a subfolder of the primary site.

Both www.example2.com and www.example.com/www.example2.com are accessible as the same content, but I want to block access the later while allowing access to the former.

Will a robots.txt file in the primary site disallowing www.* allow www.example2.com to be crawled?

Improve this question
1
  • This is not a good configuration scheme, in fact, it is horrible. No site should reside inside of another. You should really fix this. That said, Will a robots.txt file in the primary site disallowing www. allow www.example2.com to be crawled? Short answer? No. Robots.txt must always be in the root directory of a site. No exceptions.
    – closetnoc
    Commented Mar 23, 2017 at 6:16

2 Answers 2

Reset to default
2

I guess what you are looking for is a robots.txt entry like this:

User-agent: *
Disallow: /www.example2.com

Let's suggestion you have more than 100 'parked' exampleNR.com URLs, but don't want to write a line for every single one of them...use this:

User-agent: *
Disallow: /www.example

The problem is, it actually is not officially supported, but many robots like Googlebot are able to understand those easy wildcards. RegEx are definitely not supported. for additonal information

UPDATE

Deleted the trailing asterisk since robots.txt uses simple prefix matching anyway. Thanks for your attention, w3dk

Improve this answer
1
  • In your second example, there is no need for a wildcard * on the end of the URL-path since robots.txt uses simple prefix matching anyway. This "prefix matching" allows your first example to block /www.example2.com/path/to/file. All that would seem to be required is a Disallow: /www. directive in the robots.txt file for the primary domain (ie. example.com/robots.txt).
    – MrWhite
    Commented Mar 23, 2017 at 16:11
-1

From what you are saying if everything is redirected to example.com then yes.

Improve this answer
1
  • 1
    Can you please explain more - a one line answer is rarely sufficient. Whilst the answer is approximately "yes", nothing is strictly being "redirected" (although you are using this as a very lose term). The OPs suggestion of what to put in the robots.txt file is also unclear, so could be clarified in your answer.
    – MrWhite
    Commented Mar 23, 2017 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.