I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the crawler, but when we do the majority are about our handling of robots.txt. Most often the Webmaster made a mistake in his robots.txt and we kindly point out the error. But periodically we run into grey areas that involve the handling of Allow and Disallow.

The robots.txt page doesn't cover Allow. I've seen other pages, some of which say that crawlers use a "first matching" rule, and others that don't specify. That leads to some confusion. For example, Google's page about robots.txt used to have this example:

User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html

Obviously, a "first matching" rule here wouldn't work because the crawler would see the Disallow and go away, never crawling the file that was specifically allowed.

We're in the clear if we ignore all Allow lines, but then we might not crawl something that we're allowed to crawl. We'll miss things.

We've had great success by checking Allow first, and then checking Disallow, the idea being that Allow was intended to be more specific than Disallow. That's because, by default (i.e. in the absence of instructions to the contrary), all access is allowed. But then we run across something like this:

User-agent: *
Disallow: /norobots/
Allow: /

The intent here is obvious, but that Allow: / will cause a bot that checks Allow first to think it can crawl anything on the site.

Even that can be worked around in this case. We can compare the matching Allow with the matching Disallow and determine that we're not allowed to crawl anything in /norobots/. But that breaks down in the face of wildcards:

User-agent: *
Disallow: /norobots/
Allow: /*.html$

The question, then, is the bot allowed to crawl /norobots/index.html?

The "first matching" rule eliminates all ambiguity, but I often see sites that show something like the old Google example, putting the more specific Allow after the Disallow. That syntax requires more processing by the bot and leads to ambiguities that can't be resolved.

My question, then, is what's the right way to do things? What do Webmasters expect from a well-behaved bot when it comes to robots.txt handling?

  • Simply processing all the directives in order (top to bottom) would seem to remove all ambiguity (including the processing of wildcard URL patterns, which is officially "undefined"). However, the main search engine bots (Googlebot, Bingbot, etc.) don't work like that, so that's not workable.
    – MrWhite
    Oct 28, 2016 at 12:09

3 Answers 3


One very important note: the Allow statement should come before the Disallow statement, no matter how specific your statements are. So in your third example - no, the bots won't crawl /norobots/index.html.

Generally, as a personal rule, I put allow statements first and then I list the disallowed pages and folders.

  • 1
    But do you know if that's the generally accepted interpretation of robots.txt? That is, if I were to implement the "first matching rule" standard, would that be what most Webmasters expect? Dec 3, 2010 at 16:28
  • 5
    The description of Allow in the Wikipedia article on robots.txt - en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive - (and several other sites) says that "first matching rule" is the standard. Since that's 100% unambiguous and easy to implement and prove correct, that's what I went with. Dec 4, 2010 at 16:17
  • 1
    Thanks for answering your own question :) I wasn't sure how to respond. I don't know about the generally accepted method, however that's how I've been using it ever since I started working as a SEO expert and it has always worked correctly. Dec 4, 2010 at 22:07
  • 1
    The order of Allow and Disallow statements makes no difference for Googlebot which looks at specificity instead. Bing also but don't know about other search engines so if in doubt put Allows first I guess.
    – pelms
    Jun 20, 2013 at 13:46
  • 1
    This is incorrect, at least for Google. They state: 'At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined'. developers.google.com/webmasters/control-crawl-index/docs/… Best way is to test in Search Console
    – Max
    Apr 26, 2016 at 9:17

Here's my take on what I see in those three examples.

Example 1
I would ignore the entire /folder1/ directory except the myfile.html file. Since they explicitly allow it I would assume it was simply easier to block the entire directory and explicitly allow that one file as opposed to listing every file they wanted to have blocked. If that directory contained a lot of files and subdirectories that robots.txt file could get unwieldy fast.

Example 2
I would assume the /norobots/ directory is off limits and everything else is available to be crawled. I read this as "crawl everything except the /norobots/ directory".

Example 3
Similar to example 2, I would assume the /norobots/ directory is off limits and all .html files not in that directory is available to be crawled. I read this as "crawl all .html files but do not crawl any content in the the /norobots/ directory".

Hopefully your bot's user-agent contains a URL where they can find out more information about your crawling habits and make removal requests or give you feedback about how they want their robots.txt interpreted.

  • 1
    Your answers probably match the intent, except for the last one, which I find somewhat questionable. In those specific cases I can code the robots handling correctly, but there are other ambiguous cases that aren't as easily determined. More, I'm looking for a general solution. And, yes, our user-agent string has a link to our FAQ page. And although we can describe how we handle robots.txt, it'd be best for all concerned if we didn't have to. That is, if everybody handled things in the same way. But that doesn't appear to be the case. Dec 2, 2010 at 5:09
  • The above is how Googlebot handles your examples as can be tested on their robots.txt testing tool (Webmaster Tools > Blocked URLs). Google doesn't mind whether you put the Allows or Disallows first but has an algorithm that determines specificity, which leads to some not obvious results. For example, if you replace '/*.html$' with '/*myfile.html$' in the 3rd example, then 'myfile.html' is allowed rather than being blocked OR if you lose the trailing '/' from '/norobots/', .html files are also allowed.
    – pelms
    Jun 20, 2013 at 13:44
  • Aha! According to Wikipedia, Google just looks at the number of characters to determine which directive to use and in the case of a 'draw' goes with 'Allow'.
    – pelms
    Jun 20, 2013 at 13:52

Google expanded the documentation for robots.txt for user agents that support the Allow directive. The rule that Googlebot uses (and what Google is trying to make standard) is that the longest matching rule wins.

So when you have:

Disallow: /norobots/
Disallow: /nobot/
Allow:    /*.html$
Allow:    /****.gif$
  • /norobots/index.html is blocked because it matches two rules and /norobots/ is longer (10 characters) than /*.html$ (8 characters).
  • /nobot/index.html is allowed because it matches two rules and /nobot/ is shorter (7 characters) than /*.html$ (8 characters).
  • /norobots/pic.gif is allowed because it matches two rules and /norobots/ is equal length (10 characters) to /****.gif$ (10 characters). Google's spec says that the "less restrictive" rule should be used for rules of equal length, ie. the one that allows crawling.
  • I think Google's spec is a bit ambiguous... "In case of conflicting rules, including those with wildcards, the least restrictive rule is used." (previously Google has always stated that the precedence of rules with wildcards was "undefined"). I take that to mean any "wildcard" rule, regardless of length. However, in the case of wildcards, the longest matching rule still appears to win. A more specific Disallow rule with wildcards still wins over a shorter Allow rule with wildcards.
    – DocRoot
    Jul 21, 2020 at 17:53
  • /****.gif$ - it gets a bit "confusing" when using multiple adjacent wildcards since * matches "0 or more" anyway, so /****.gif$ is really the same as /*.gif$ - however, it appears to have different precedence.
    – DocRoot
    Jul 21, 2020 at 17:56
  • Yeah the multiple stars are there just to increase the length of the rule and it's precedence. Jul 21, 2020 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.