1

I have a robots.txt on my www site. Do I also need one on other sites like my app and forums sites?

Alexa audit keeps finding bad links, missing tags, and other "errors" on my vanilla forums site even though the www site's robots.txt says to ignore specific directories that are on my app and forum sites.

1
  • It's not just Alexa, every crawler needs a robots.txt on each and every subdomain on which you want to control crawling. – Stephen Ostermiller Sep 12 '17 at 20:33
2

The first thing you have to remember is that example.com, www.example.com, sub-domain.example.com, even HTTP versus HTTPS are all different sites.

Robots.txt cannot reference other sites, it is only for the site it resides upon. So for any sub-domain you will need a robots.txt just for that site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.