In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between these two rules:

User-agent: *
Disallow:

and this:

User-agent: *
Allow: /

Thank you

link|improve this question
feedback

migrated from stackoverflow.com Sep 22 '11 at 15:38

This question came from our site for professional and enthusiast programmers.

2 Answers

Allow is non-standard according to http://en.wikipedia.org/wiki/Robots.txt

link|improve this answer
feedback

User-agent: * Disallow: /

The one above tells robots not to crawl anything that matches the url path. With the above instruction, Googlebot other search engine bots will not crawl your entire website.

User-agent: * Allow: /

The one above tells everything is allowed. All visitng bots including Googlebot is allowed to crawl the website.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown