2

If I place a regular expression in robots.txt will it be readable for the web crawlers?

8

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.

4

While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching.

For example:

User-agent: Googlebot
Disallow: /private*/

Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports * and $ (end of string).

Details on Googlebot's pattern matching: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

  • Note that Disallow: /private*/ does not block "any directory", but any URLs whose path begins with this string. – unor Oct 19 '13 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.