1
user-agent: *
disallow:

Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?

3

Yes. The wild card * after user agent means it applies to all crawlers and as there is nothing defined after the Disallow, the entire site is free to be crawled.

3

If you want to allow access to all content of your website, just don't create a robots.txt, or create a blank one (to avoid 404 errors).

  • 1
    robots.txt is still a handy file to have, especially for listing sitemap files. – Andrew Lott Jul 1 '13 at 14:16
  • I need it to declare my sitemaps... – Jérôme Verstrynge Jul 1 '13 at 14:36
  • 1
    I think managing sitemap.xml can be done out of a robots.txt. If you need to declare your sitemap, please edit your question. – Zistoloen Jul 1 '13 at 14:37
  • 1
    It's useful to have a robots.txt file just to prevent your server error log file from filling up with 404 requests for robots.txt! – AllInOne Jul 1 '13 at 22:55
3

To use robots.txt and the functionality in a nutshell: http://www.robotstxt.org/robotstxt.html.

As for a sitemap.xml / sitemap.xml.gz you don't need specific a robots.txt. If i.e. Google is allowed (and so to see you want that, why else Disallow without any) to crawl the site you can give Google the exact path. (create Google account and use Webmaster Tools, it is free and imho usefull. It even has option to create robots.txt for you)

Anyway, still want to use the robots.txt and sitemap path then this could be sample:

User-agent: *
Disallow: 
sitemap: http://name of your website/sitemap.xml

Be aware that robots.txt file is publicly available so don't use it with the intention to hide information. A .htaccess file is a much more better please to hide/protect "stuff". (if there is no access on/at server level itself)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.