Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top
user-agent: *

Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?

share|improve this question
up vote 3 down vote accepted

Yes. The wild card * after user agent means it applies to all crawlers and as there is nothing defined after the Disallow, the entire site is free to be crawled.

share|improve this answer

If you want to allow access to all content of your website, just don't create a robots.txt, or create a blank one (to avoid 404 errors).

share|improve this answer
robots.txt is still a handy file to have, especially for listing sitemap files. – Andrew Lott Jul 1 '13 at 14:16
I need it to declare my sitemaps... – JVerstry Jul 1 '13 at 14:36
I think managing sitemap.xml can be done out of a robots.txt. If you need to declare your sitemap, please edit your question. – Zistoloen Jul 1 '13 at 14:37
It's useful to have a robots.txt file just to prevent your server error log file from filling up with 404 requests for robots.txt! – AllInOne Jul 1 '13 at 22:55

To use robots.txt and the functionality in a nutshell: http://www.robotstxt.org/robotstxt.html.

As for a sitemap.xml / sitemap.xml.gz you don't need specific a robots.txt. If i.e. Google is allowed (and so to see you want that, why else Disallow without any) to crawl the site you can give Google the exact path. (create Google account and use Webmaster Tools, it is free and imho usefull. It even has option to create robots.txt for you)

Anyway, still want to use the robots.txt and sitemap path then this could be sample:

User-agent: *
sitemap: http://name of your website/sitemap.xml

Be aware that robots.txt file is publicly available so don't use it with the intention to hide information. A .htaccess file is a much more better please to hide/protect "stuff". (if there is no access on/at server level itself)

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.