Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

If I place a regular expression in robots.txt will it be readable for the web crawlers?

share|improve this question

2 Answers 2

up vote 8 down vote accepted

No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html.

share|improve this answer
    
Wildcards are not supported by the original robots.txt specification. –  unor Oct 19 '13 at 20:40

While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching.

For example:

User-agent: Googlebot
Disallow: /private*/

Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports * and $ (end of string).

Details on Googlebot's pattern matching: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

share|improve this answer
    
Note that Disallow: /private*/ does not block "any directory", but any URLs whose path begins with this string. –  unor Oct 19 '13 at 20:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.