Please help me to understand the following code in robots.txt:
Allow: /
Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this communityAllow is not officially part of the robots.txt specifications but the major search engines do honor it. But it is redundant as anything not explicitly blocked is allowed to be crawled.
It is possible to use the Allow
directive after a Disallow
directive to allow crawlers to visit URLs that would otherwise be blocked by the Disallow
directive. See: Is it possible to block an entire site to Google then list exceptions?
As John says, the Allow
directive is not standard, but it is supported by the major search engine spiders like Googlebot.