user-agent: *
disallow:
Is this the right way to allow all access to all content of a website to crawlers in a robots.txt
?
Yes. The wild card *
after user agent means it applies to all crawlers and as there is nothing defined after the Disallow
, the entire site is free to be crawled.
If you want to allow access to all content of your website, just don't create a robots.txt, or create a blank one (to avoid 404 errors).
To use robots.txt and the functionality in a nutshell: http://www.robotstxt.org/robotstxt.html.
As for a sitemap.xml / sitemap.xml.gz you don't need specific a robots.txt.
If i.e. Google is allowed (and so to see you want that, why else Disallow
without any) to crawl the site you can give Google the exact path. (create Google account and use Webmaster Tools, it is free and imho usefull. It even has option to create robots.txt for you)
Anyway, still want to use the robots.txt and sitemap path then this could be sample:
User-agent: *
Disallow:
sitemap: http://name of your website/sitemap.xml
Be aware that robots.txt file is publicly available so don't use it with the intention to hide information. A .htaccess file is a much more better please to hide/protect "stuff". (if there is no access on/at server level itself)