5

I have a robots.txt file:

User-agent:*
Disallow:/path/page
Disallow:/path/
Allow:/

The disallowed path is still getting crawled.

I have searched this problem and what they said, order of precedence in google does not matter. So technically, the disallowed should work but now I'm wondering if it is because Allow:/ is overriding it?

  • 2
    robotstxt.org/robotstxt.html You do not have an agents directive. As well, you do not have to specify what you allow, but do have to specify what you disallow. – closetnoc Oct 25 '16 at 16:39
  • 2
    Sorry I didn't put it there but yes I do have User-agent:*. Yes I was wondering about that. I know by default, all web-crawlers are allowed. I think what was happening is that because I have "Allow:/", it's like overwriting it? Is this correct? – user71368 Oct 25 '16 at 16:49
  • "order of precedence in google does not matter" - you mean the "order of the directives does not matter". There is always an order of precedence (unless you are using "wildcards", in which case it is officially "undefined"). – MrWhite Oct 25 '16 at 20:15
  • 1
    As others have stated, you don't need the Allow: directive, but neither should this be causing problems. The other "disallowed" URLs should still be blocked regardless of whether the Allow: directive is present or not. What does GSC report? When did you make this change? Do you have other directives in your robots.txt file? You've presumably confirmed from your logs that Googlebot is still crawling these URLs? – MrWhite Oct 25 '16 at 20:29
5

Putting your page through Google's robots.txt tester reveals two problems:

  1. You have no User-agent line so your rules don't apply to any crawler: enter image description here
  2. Once you put in the User-agent line, the Allow line overrides the disallow: enter image description here

The correct robots.txt file would be:

User-agent: *
Disallow: /path/page
Disallow: /path/

DO NOT use Allow:. Allowing crawling is the default. You only need to include the items you don't want crawled.

Include the User-Agent: line to specify that it applies to all crawlers. Otherwise, it will apply to none.

I don't think that having a space after the colon actually matters, but all the examples I see have it.

I should also add that robots.txt is only for crawlers that choose to obey it. Search engine spiders like Googlebot should obey robots.txt, however not all other spiders will.

  • 2
    "page/path" - You seem to be testing the wrong URL-path. If you tested "/path/page" as stated in the Disallow directive then this should be "BLOCKED" - since the most specific rule should win. – MrWhite Oct 25 '16 at 20:11
2

As already mentioned, the Allow: directive in this instance is superfluous. The default action is to allow all crawling, so explicitly stating Allow: / (ie. allow all) is entirely redundant.

However, contrary to what has been suggested, neither would the Allow: / directive cause you any problems. The Allow: / directive will not "override" other Disallow: directives, because it is the least specific, regardless of the apparent order.

order of precedence in google does not matter.

Yes, sort of. You mean the "order of the directives does not matter". There is always an order of precedence (unless you are using "wildcards", in which case it is officially "undefined"). This is why the Allow: / directive does not override the more specific Disallow: directives above it. Google defines the order of precedence:

for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule.

And this is confirmed by using Google's robots.txt Tester, when testing a disallowed path eg. /path/page:

Google's robot.txt Tester - Disallow

This is at least how Googlebot and Bingbot work (the most specific path wins). However, some (old) bots reportedly use a "first match" rule. So, for greatest compatibility it is recommended to include any Allow: directives first. Reference: What's the proper way to handle Allow and Disallow in robots.txt?

Also, since robots.txt is prefix matching, the Disallow: /path/page directive is also superfluous, since Disallow: /path/ will block /path/page as well. So, in summary, your robots.txt file only needs the one Disallow directive, the others are simply superfluous but will not actually cause any harm:

User-agent: *
Disallow: /path/

White space before the path is entirely optional, although as noted in Stephen's answer, it is much more common to see it and it arguably makes it more readable.

The only time you would need an Allow: directive is if you need to make an exception and allow a URL that would otherwise be blocked by a Disallow: directive. eg. If you wanted to allow /path/foo in the above robots.txt file then you would need to explicitly include the Allow: /path/foo directive somewhere in the group.

The disallowed path is still getting crawled.

If this is still the case, then something else is going on...

  • Do you have any other directives in your robots.txt file. Test the URL in Google's robot.txt Tester.
  • When was the current robots.txt file implemented? Google only picks up changes to the robots.txt file every day or so. In GSC you can identify which version Googlebot is currently using.
  • As Stephen has already pointed out, robots.txt is only honoured by the "good bots". Many (bad) bots will simply ignore it and crawl your URLs regardless. You can check in your access logs as to whether the "good" bots are still crawling these disallowed URLs.
  • 1
    I see what you mean in your comment. I had elements in the path mixed up. Thank you for posting your own answer addressing it further. – Stephen Ostermiller Oct 27 '16 at 20:48
  • @StephenOstermiller There seems to be something about "/path/page" that makes it easy to type the wrong way round (in a tongue twister sort of way?) - I initially typed it the wrong way round a couple of times as well!? – MrWhite Oct 28 '16 at 0:01
0

It is not clear from your question what kind of site you have. If it is a SPA, it would explain the problems that you are having. robots.txt does not seem to deal effectively with SPA's and other solutions need to be looked for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.