2

Please help me to understand the following code in robots.txt:

Allow: /
3

Allow is not officially part of the robots.txt specifications but the major search engines do honor it. But it is redundant as anything not explicitly blocked is allowed to be crawled.

3

It is possible to use the Allow directive after a Disallow directive to allow crawlers to visit URLs that would otherwise be blocked by the Disallow directive. See: Is it possible to block an entire site to Google then list exceptions?

As John says, the Allow directive is not standard, but it is supported by the major search engine spiders like Googlebot.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.