If I don't want to set any special behavior, is it OK if I don't bother to have a robots.txt file?
Or can the lack of one be harmful?
Add a comment
|
6 Answers
Lack of a robots.txt file will not be harmful. From the robotstxt.org website:
To allow all robots complete access
User-agent: *
Disallow:(or just create an empty "/robots.txt" file, or don't use one at all)
However, even if you are not specifying anything in your robots.txt file, it is a good way of informing search engines of the location of your XML Sitemap. You can do this by adding a line at the top of your robots.txt file that looks something like:
You should also be aware that not having it will create a lot of 404 entries in your web logs.
-
+1 - good summary, though I'd really emphasize what Kinopiko has correctly stressed already: just create the most basic or even an empty one to avoid those 404s and (depending on your sites 404 page handling) potentially quite some traffic/bandwidth, as search engines will apply proper HTTP cache control to skip downloading the file again if unchanged, be it empty or not. Jul 10, 2010 at 11:02
-
If you don't have a "robots.txt" your error log will get lots of 404s on the file, which could be a kind of annoyance, similar to if you don't have a favicon.
-
2
-
2+1 - I'd like to add that you don't just save yourself from larger and noisy log files, but can (depending on your sites 404 page handling) avoid potentially quite some traffic/bandwidth as well due to most 404 pages being larger than a simple
robots.txtfile, which in addition will be downloaded less often due to search engines applying proper HTTP cache control. Jul 10, 2010 at 11:00
I think it would have to be OK, otherwise huge swaths of the web would be un-indexable by web spiders.
No robots.txt is the same as an "allow indexing by everyone" robots.txt almost by definition.
answered Jul 9, 2010 at 8:32
The lack of a robots.txt file leaves it up to the crawler to decide what it can and can not do. Since it takes only seconds to avoid any kind of ambiguity, why not just make one that allows all agents to access everything?
Well, since robots.txt contains the address of your sitemap, not having one is potentially harmful.
-
sitemaps are only helpful for certain types of websites though, IMO Jul 9, 2010 at 10:55
-
I've also seen crawlers (google in particular) look for /sitemap.xml or /sitemap.gz in the absence of robots.txt– Tim PostJul 9, 2010 at 12:03
-
You don't need to have your sitemap in your robots.txt, you can submit it to Google/Yahoo/Bing anyway. It's definitely not "potentially harmful". Jul 12, 2010 at 16:24
Depending on your content there should be no issues with not having a robots file as long as you hapy to have every page on your site indexed by search engines.
