Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between these two rules:

User-agent: *
Disallow:

and this:

User-agent: *
Allow: /
share|improve this question

migrated from stackoverflow.com Sep 22 '11 at 15:38

This question came from our site for professional and enthusiast programmers.

2 Answers 2

Allow is non-standard according to Wikipedia: http://en.wikipedia.org/wiki/Robots.txt.

share|improve this answer
User-agent: *
Disallow: /

The one above tells robots not to crawl anything that matches the URL path. With the above instruction, Googlebot other search engine bots will not crawl your entire website.

User-agent: *
Allow: /

The one above tells everything is allowed. All visiting bots including Googlebot is allowed to crawl the website.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.