Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?

share|improve this question

If you don't have a robots.txt your website will get indexed. The robots.txt file are just some directions for (nice) crawlers on what not to crawl.


Please also note that there are aggressive crawlers out there that just ignore it.

share|improve this answer

You may also find 404's in your error logs if you don't have a robots.txt in place, if you want to avoid those just create a blank file named robots.txt in your sites root.

share|improve this answer

A robots.txt file can also be used to provide a link to the sitemap.xml file. That file should contain links to all the pages on your site.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.