I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the crawler, but when we do the majority are about our handling of robots.txt. Most often the Webmaster made a mistake in his robots.txt and we kindly point out the error. But periodically we run into grey areas that involve the handling of
The robots.txt page doesn't cover
Allow. I've seen other pages, some of which say that crawlers use a "first matching" rule, and others that don't specify. That leads to some confusion. For example, Google's page about robots.txt used to have this example:
User-agent: Googlebot Disallow: /folder1/ Allow: /folder1/myfile.html
Obviously, a "first matching" rule here wouldn't work because the crawler would see the
Disallow and go away, never crawling the file that was specifically allowed.
We're in the clear if we ignore all
Allow lines, but then we might not crawl something that we're allowed to crawl. We'll miss things.
We've had great success by checking
Allow first, and then checking
Disallow, the idea being that
Allow was intended to be more specific than
Disallow. That's because, by default (i.e. in the absence of instructions to the contrary), all access is allowed. But then we run across something like this:
User-agent: * Disallow: /norobots/ Allow: /
The intent here is obvious, but that
Allow: / will cause a bot that checks
Allow first to think it can crawl anything on the site.
Even that can be worked around in this case. We can compare the matching
Allow with the matching
Disallow and determine that we're not allowed to crawl anything in /norobots/. But that breaks down in the face of wildcards:
User-agent: * Disallow: /norobots/ Allow: /*.html$
The question, then, is the bot allowed to crawl
The "first matching" rule eliminates all ambiguity, but I often see sites that show something like the old Google example, putting the more specific
Allow after the
Disallow. That syntax requires more processing by the bot and leads to ambiguities that can't be resolved.
My question, then, is what's the right way to do things? What do Webmasters expect from a well-behaved bot when it comes to robots.txt handling?