Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between these two rules:

User-agent: *

and this:

User-agent: *
Allow: /
share|improve this question

migrated from stackoverflow.com Sep 22 '11 at 15:38

This question came from our site for professional and enthusiast programmers.

Allow is non-standard according to Wikipedia: http://en.wikipedia.org/wiki/Robots.txt.

share|improve this answer
User-agent: *
Disallow: /

The one above tells robots not to crawl anything that matches the URL path. With the above instruction, Googlebot other search engine bots will not crawl your entire website.

User-agent: *
Allow: /

The one above tells everything is allowed. All visiting bots including Googlebot is allowed to crawl the website.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.