1

In our robots.txt we have something like

User-agent: *
Disallow: /v6.0
Disallow: /v5.8
Disallow: /v5.7
Disallow: /v5.5
Disallow: /v5.1
Disallow: /v5.0

Which properly prevents search engines from crawling any of our URLs prefixed by /v6.0, /v5.8 etc... However, there are certain pages within those disallowed URLs that we do want indexed, such as /v6.0/i-want-this-page-indexable

Is doing such a thing possible? If so, what would be the appropriate syntax?

2

To allow a more specific URL you can use the Allow directive (supported by all major search engine bots). For example:

User-agent: *
Allow: /v6.0/i-want-this-page-indexable
Disallow: /v6.0
Disallow: /v5.8
Disallow: /v5.7
Disallow: /v5.5
Disallow: /v5.1
Disallow: /v5.0

The order of these directives doesn't strictly matter for most search engine bots (including Google) as the most specific URL-path (by length) will win. However, for some search engine bots the first matching URL-path wins, so the Allow directive should go before the Disallow directive where there is a conflicting URL-path.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.