Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
|
|
||||
|
|
|
If you don't have a robots.txt your website will get indexed. The robots.txt file are just some directions for (nice) crawlers on what not to crawl. http://www.robotstxt.org/robotstxt.html Please also note that there are aggressive crawlers out there that just ignore it. |
|||
|
|
|
You may also find 404's in your error logs if you don't have a robots.txt in place, if you want to avoid those just create a blank file named robots.txt in your sites root. |
|||
|
|
|
A robots.txt file can also be used to provide a link to the sitemap.xml file. That file should contain links to all the pages on your site. |
|||
|
|
