Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If I place a regular expression in robots.txt will it be readable for the web crawlers?

share|improve this question
up vote 8 down vote accepted

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.

share|improve this answer
Wildcards are not supported by the original robots.txt specification. – unor Oct 19 '13 at 20:40

While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching.

For example:

User-agent: Googlebot
Disallow: /private*/

Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports * and $ (end of string).

Details on Googlebot's pattern matching: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

share|improve this answer
Note that Disallow: /private*/ does not block "any directory", but any URLs whose path begins with this string. – unor Oct 19 '13 at 20:45

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.