Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?

share|improve this question

3 Answers 3

If you don't have a robots.txt your website will get indexed. The robots.txt file are just some directions for (nice) crawlers on what not to crawl.

http://www.robotstxt.org/robotstxt.html

Please also note that there are aggressive crawlers out there that just ignore it.

share|improve this answer

You may also find 404's in your error logs if you don't have a robots.txt in place, if you want to avoid those just create a blank file named robots.txt in your sites root.

share|improve this answer

A robots.txt file can also be used to provide a link to the sitemap.xml file. That file should contain links to all the pages on your site.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.