Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm very confused about Google's robots.txt documentation

Examples of valid robots.txt URLs:

Not valid for
http://other.example.com/
https://example.com/
http://example.com:8181/

It says the robots.txt is not valid for subdomains. Is that correct? If yes, what if we have different contents with main domain?

share|improve this question
1  
You seem to have misquoted (or misunderstood) the docs... that table states that the particular robots.txt file located at http://example.com/robots.txt is not valid for the URLs you have quoted above, not that those URLs are invalid locations for the robots.txt file. –  w3d Feb 26 at 16:46
add comment

1 Answer 1

up vote 3 down vote accepted

You need to put a robots.txt in the root web directory of your subdomain, just like you do for your main domain. You have to think of the subdomain as a separate website in this case.

share|improve this answer
    
Yes, i have a robots.txt on the web root directory of subdomain. So, the google documentation is wrong? –  richard Feb 26 at 15:41
2  
They are correct. What they are saying is the robots.txt file for the main domain will not affect the subdomain. The subdomain needs its own robots.txt file. –  John Conde Feb 26 at 15:42
    
Thanks a lot! it's clear now. :) –  richard Feb 26 at 15:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.