Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?

3 Answers 3


If you don't have a robots.txt your website will get indexed. The robots.txt file are just some directions for (nice) crawlers on what not to crawl.


Please also note that there are aggressive crawlers out there that just ignore it.


You may also find 404's in your error logs if you don't have a robots.txt in place, if you want to avoid those just create a blank file named robots.txt in your sites root.


A robots.txt file can also be used to provide a link to the sitemap.xml file. That file should contain links to all the pages on your site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.