As already mentioned, the
Allow: directive in this instance is superfluous. The default action is to allow all crawling, so explicitly stating
Allow: / (ie. allow all) is entirely redundant.
However, contrary to what has been suggested, neither would the
Allow: / directive cause you any problems. The
Allow: / directive will not "override" other
Disallow: directives, because it is the least specific, regardless of the apparent order.
order of precedence in google does not matter.
Yes, sort of. You mean the "order of the directives does not matter". There is always an order of precedence (unless you are using "wildcards", in which case it is officially "undefined"). This is why the
Allow: / directive does not override the more specific
Disallow: directives above it. Google defines the order of precedence:
for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule.
And this is confirmed by using Google's robots.txt Tester, when testing a disallowed path eg.
This is at least how Googlebot and Bingbot work (the most specific path wins). However, some (old) bots reportedly use a "first match" rule. So, for greatest compatibility it is recommended to include any
Allow: directives first. Reference: What's the proper way to handle Allow and Disallow in robots.txt?
Also, since robots.txt is prefix matching, the
Disallow: /path/page directive is also superfluous, since
Disallow: /path/ will block
/path/page as well. So, in summary, your
robots.txt file only needs the one
Disallow directive, the others are simply superfluous but will not actually cause any harm:
White space before the path is entirely optional, although as noted in Stephen's answer, it is much more common to see it and it arguably makes it more readable.
The only time you would need an
Allow: directive is if you need to make an exception and allow a URL that would otherwise be blocked by a
Disallow: directive. eg. If you wanted to allow
/path/foo in the above
robots.txt file then you would need to explicitly include the
Allow: /path/foo directive somewhere in the group.
The disallowed path is still getting crawled.
If this is still the case, then something else is going on...
- Do you have any other directives in your
robots.txt file. Test the URL in Google's robot.txt Tester.
- When was the current
robots.txt file implemented? Google only picks up changes to the
robots.txt file every day or so. In GSC you can identify which version Googlebot is currently using.
- As Stephen has already pointed out,
robots.txt is only honoured by the "good bots". Many (bad) bots will simply ignore it and crawl your URLs regardless. You can check in your access logs as to whether the "good" bots are still crawling these disallowed URLs.
Allow:directive, but neither should this be causing problems. The other "disallowed" URLs should still be blocked regardless of whether the
Allow:directive is present or not. What does GSC report? When did you make this change? Do you have other directives in your robots.txt file? You've presumably confirmed from your logs that Googlebot is still crawling these URLs?