3

If I place a regular expression in robots.txt will it be readable for the web crawlers?

Improve this question

2 Answers 2

Reset to default
10

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.

Improve this answer
1
6

While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching.

For example:

User-agent: Googlebot
Disallow: /private*/

Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports * and $ (end of string).

Details on Googlebot's pattern matching: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

Improve this answer
1
  • Note that Disallow: /private*/ does not block "any directory", but any URLs whose path begins with this string.
    – unor
    Oct 19, 2013 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.