# If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

Or can the lack of one be harmful?

Lack of a robots.txt file will not be harmful. From the robotstxt.org website:

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

However, even if you are not specifying anything in your robots.txt file, it is a good way of informing search engines of the location of your XML Sitemap. You can do this by adding a line at the top of your robots.txt file that looks something like:

You should also be aware that not having it will create a lot of 404 entries in your web logs.

• +1 - good summary, though I'd really emphasize what Kinopiko has correctly stressed already: just create the most basic or even an empty one to avoid those 404s and (depending on your sites 404 page handling) potentially quite some traffic/bandwidth, as search engines will apply proper HTTP cache control to skip downloading the file again if unchanged, be it empty or not. – Steffen Opel Jul 10 '10 at 11:02
• Marked as wiki, please edit as you see fit. – JasonBirch Jul 11 '10 at 4:08

If you don't have a "robots.txt" your error log will get lots of 404s on the file, which could be a kind of annoyance, similar to if you don't have a favicon.

• an excellent point.. – Jeff Atwood Jul 9 '10 at 10:54
• +1 - I'd like to add that you don't just save yourself from larger and noisy log files, but can (depending on your sites 404 page handling) avoid potentially quite some traffic/bandwidth as well due to most 404 pages being larger than a simple robots.txt file, which in addition will be downloaded less often due to search engines applying proper HTTP cache control. – Steffen Opel Jul 10 '10 at 11:00

I think it would have to be OK, otherwise huge swaths of the web would be un-indexable by web spiders.

No robots.txt is the same as an "allow indexing by everyone" robots.txt almost by definition.

The lack of a robots.txt file leaves it up to the crawler to decide what it can and can not do. Since it takes only seconds to avoid any kind of ambiguity, why not just make one that allows all agents to access everything?

Well, since robots.txt contains the address of your sitemap, not having one is potentially harmful.

• sitemaps are only helpful for certain types of websites though, IMO – Jeff Atwood Jul 9 '10 at 10:55
• I've also seen crawlers (google in particular) look for /sitemap.xml or /sitemap.gz in the absence of robots.txt – Tim Post Jul 9 '10 at 12:03
• You don't need to have your sitemap in your robots.txt, you can submit it to Google/Yahoo/Bing anyway. It's definitely not "potentially harmful". – DisgruntledGoat Jul 12 '10 at 16:24

Depending on your content there should be no issues with not having a robots file as long as you hapy to have every page on your site indexed by search engines.