Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 22 down vote favorite
7

If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?

Or can the lack of one be harmful?

share|improve this question

6 Answers

up vote 26 down vote accepted

Lack of a robots.txt file will not be harmful. From the robotstxt.org website:

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

However, even if you are not specifying anything in your robots.txt file, it is a good way of informing search engines of the location of your XML Sitemap. You can do this by adding a line at the top of your robots.txt file that looks something like:

Sitemap: http://www.example.com/sitemap-host1.xml

You should also be aware that not having it will create a lot of 404 entries in your web logs.

share|improve this answer
+1 - good summary, though I'd really emphasize what Kinopiko has correctly stressed already: just create the most basic or even an empty one to avoid those 404s and (depending on your sites 404 page handling) potentially quite some traffic/bandwidth, as search engines will apply proper HTTP cache control to skip downloading the file again if unchanged, be it empty or not. – Steffen Opel Jul 10 '10 at 11:02
Marked as wiki, please edit as you see fit. – JasonBirch Jul 11 '10 at 4:08
up vote 19 down vote

If you don't have a "robots.txt" your error log will get lots of 404s on the file, which could be a kind of annoyance, similar to if you don't have a favicon.

share|improve this answer
1  
an excellent point.. – Jeff Atwood♦ Jul 9 '10 at 10:54
1  
+1 - I'd like to add that you don't just save yourself from larger and noisy log files, but can (depending on your sites 404 page handling) avoid potentially quite some traffic/bandwidth as well due to most 404 pages being larger than a simple robots.txt file, which in addition will be downloaded less often due to search engines applying proper HTTP cache control. – Steffen Opel Jul 10 '10 at 11:00
up vote 6 down vote

I think it would have to be OK, otherwise huge swaths of the web would be un-indexable by web spiders.

No robots.txt is the same as an "allow indexing by everyone" robots.txt almost by definition.

share|improve this answer
up vote 2 down vote

The lack of a robots.txt file leaves it up to the crawler to decide what it can and can not do. Since it takes only seconds to avoid any kind of ambiguity, why not just make one that allows all agents to access everything?

share|improve this answer
up vote 0 down vote

Well, since robots.txt contains the address of your sitemap, not having one is potentially harmful.

share|improve this answer
sitemaps are only helpful for certain types of websites though, IMO – Jeff Atwood♦ Jul 9 '10 at 10:55
I've also seen crawlers (google in particular) look for /sitemap.xml or /sitemap.gz in the absence of robots.txt – Tim Post♦ Jul 9 '10 at 12:03
You don't need to have your sitemap in your robots.txt, you can submit it to Google/Yahoo/Bing anyway. It's definitely not "potentially harmful". – DisgruntledGoat Jul 12 '10 at 16:24
up vote 0 down vote

Depending on your content there should be no issues with not having a robots file as long as you hapy to have every page on your site indexed by search engines.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.