Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
user-agent: *
disallow:

Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

Yes. The wild card * after user agent means it applies to all crawlers and as there is nothing defined after the Disallow, the entire site is free to be crawled.

share|improve this answer
add comment

If you want to allow access to all content of your website, just don't create a robots.txt, or create a blank one (to avoid 404 errors).

share|improve this answer
1  
robots.txt is still a handy file to have, especially for listing sitemap files. –  Andrew Lott Jul 1 '13 at 14:16
    
I need it to declare my sitemaps... –  JVerstry Jul 1 '13 at 14:36
1  
I think managing sitemap.xml can be done out of a robots.txt. If you need to declare your sitemap, please edit your question. –  Zistoloen Jul 1 '13 at 14:37
1  
It's useful to have a robots.txt file just to prevent your server error log file from filling up with 404 requests for robots.txt! –  AllInOne Jul 1 '13 at 22:55
add comment

To use robots.txt and the functionality in a nutshell: http://www.robotstxt.org/robotstxt.html.

As for a sitemap.xml / sitemap.xml.gz you don't need specific a robots.txt. If i.e. Google is allowed (and so to see you want that, why else Disallow without any) to crawl the site you can give Google the exact path. (create Google account and use Webmaster Tools, it is free and imho usefull. It even has option to create robots.txt for you)

Anyway, still want to use the robots.txt and sitemap path then this could be sample:

User-agent: *
Disallow: 
sitemap: http://name of your website/sitemap.xml

Be aware that robots.txt file is publicly available so don't use it with the intention to hide information. A .htaccess file is a much more better please to hide/protect "stuff". (if there is no access on/at server level itself)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.