If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

Or can the lack of one be harmful?

6 Answers 6


Lack of a robots.txt file will not be harmful. From the robotstxt.org website:

To allow all robots complete access

User-agent: *

(or just create an empty "/robots.txt" file, or don't use one at all)

However, even if you are not specifying anything in your robots.txt file, it is a good way of informing search engines of the location of your XML Sitemap. You can do this by adding a line at the top of your robots.txt file that looks something like:

Sitemap: http://www.example.com/sitemap-host1.xml

You should also be aware that not having it will create a lot of 404 entries in your web logs.

  • +1 - good summary, though I'd really emphasize what Kinopiko has correctly stressed already: just create the most basic or even an empty one to avoid those 404s and (depending on your sites 404 page handling) potentially quite some traffic/bandwidth, as search engines will apply proper HTTP cache control to skip downloading the file again if unchanged, be it empty or not. Commented Jul 10, 2010 at 11:02
  • Marked as wiki, please edit as you see fit.
    – JasonBirch
    Commented Jul 11, 2010 at 4:08

If you don't have a "robots.txt" your error log will get lots of 404s on the file, which could be a kind of annoyance, similar to if you don't have a favicon.

  • 2
    an excellent point.. Commented Jul 9, 2010 at 10:54
  • 2
    +1 - I'd like to add that you don't just save yourself from larger and noisy log files, but can (depending on your sites 404 page handling) avoid potentially quite some traffic/bandwidth as well due to most 404 pages being larger than a simple robots.txt file, which in addition will be downloaded less often due to search engines applying proper HTTP cache control. Commented Jul 10, 2010 at 11:00

I think it would have to be OK, otherwise huge swaths of the web would be un-indexable by web spiders.

No robots.txt is the same as an "allow indexing by everyone" robots.txt almost by definition.


The lack of a robots.txt file leaves it up to the crawler to decide what it can and can not do. Since it takes only seconds to avoid any kind of ambiguity, why not just make one that allows all agents to access everything?


Well, since robots.txt contains the address of your sitemap, not having one is potentially harmful.

  • sitemaps are only helpful for certain types of websites though, IMO Commented Jul 9, 2010 at 10:55
  • I've also seen crawlers (google in particular) look for /sitemap.xml or /sitemap.gz in the absence of robots.txt
    – Tim Post
    Commented Jul 9, 2010 at 12:03
  • You don't need to have your sitemap in your robots.txt, you can submit it to Google/Yahoo/Bing anyway. It's definitely not "potentially harmful". Commented Jul 12, 2010 at 16:24

Depending on your content there should be no issues with not having a robots file as long as you hapy to have every page on your site indexed by search engines.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.