3

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?

4

If you don't have a robots.txt your website will get indexed. The robots.txt file are just some directions for (nice) crawlers on what not to crawl.

http://www.robotstxt.org/robotstxt.html

Please also note that there are aggressive crawlers out there that just ignore it.

| improve this answer | |
2

You may also find 404's in your error logs if you don't have a robots.txt in place, if you want to avoid those just create a blank file named robots.txt in your sites root.

| improve this answer | |
2

A robots.txt file can also be used to provide a link to the sitemap.xml file. That file should contain links to all the pages on your site.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.