If I place a regular expression in robots.txt will it be readable for the web crawlers?
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
|
|
|
No, the robots.txt syntax only supports wildcards, and not regular expressions. See here for a good discussion of the syntax: http://www.robotstxt.org/robotstxt.html. |
|||
|
|
|
While it's not part of the standard robots.txt protocol (and therefore not globally recognized), Google and Bing both support LIMITED pattern matching. For example: User-agent: Googlebot Disallow: /private*/ Will block Googlebot from any directory beginning with "private". Currently, Googlebot supports * and $ (end of string) Details on Googlebot's pattern matching: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 |
|||
|
|
