If I place a regular expression in robots.txt will it be readable for the web crawlers?
No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.
1Wildcards are not supported by the original robots.txt specification.– unorOct 19, 2013 at 20:40
While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching.
User-agent: Googlebot Disallow: /private*/
Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports
$ (end of string).
Details on Googlebot's pattern matching: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
Disallow: /private*/does not block "any directory", but any URLs whose path begins with this string.– unorOct 19, 2013 at 20:45