Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm very confused about Google's robots.txt documentation

Examples of valid robots.txt URLs:

Not valid for

It says the robots.txt is not valid for subdomains. Is that correct? If yes, what if we have different contents with main domain?

share|improve this question
You seem to have misquoted (or misunderstood) the docs... that table states that the particular robots.txt file located at http://example.com/robots.txt is not valid for the URLs you have quoted above, not that those URLs are invalid locations for the robots.txt file. – w3dk Feb 26 '14 at 16:46
up vote 3 down vote accepted

You need to put a robots.txt in the root web directory of your subdomain, just like you do for your main domain. You have to think of the subdomain as a separate website in this case.

share|improve this answer
Yes, i have a robots.txt on the web root directory of subdomain. So, the google documentation is wrong? – richard Feb 26 '14 at 15:41
They are correct. What they are saying is the robots.txt file for the main domain will not affect the subdomain. The subdomain needs its own robots.txt file. – John Conde Feb 26 '14 at 15:42
Thanks a lot! it's clear now. :) – richard Feb 26 '14 at 15:44

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.