3

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get redirected to dynamically-created pages under /doc. These dynamic pages are built from a database, and contain links to other /doc/ resources, so in general we don't want them to be crawled. Our robots.txt contains:

Disallow: /doc

However, we do want the non-redirected pages under /id to get indexed by Google et al:

Allow: /id

So the question I have, which I can't find an answer to so far, is: if an allowed /id page 303-redirects to a /doc page, will it still be blocked by robots.txt?

If yes, we're OK, but otherwise I'm going to disallow all /id resources in the robots file, as having the crawler hammer the db would be worse than losing search indexing for the /id pages.

Improve this question
2
  • Is it possible to make the database-generated pages under /doc/ be put at /id/ rather? That way, you can index what you want, and block what you don't want public. I don't see 303 as a good idea, though: sharkseo.com/nohat/303-redirects-seo
    – ionFish
    Commented May 17, 2012 at 1:38
  • The whole point of separating /doc and /id is so that they are different. This allows you to make the distinction between information about a thing and a reference to the thing itself. As for user agents not recognising 303 redirects, that hasn't proved to be a problem in practice, and a primary aim anyway is to support machine processing of semantic web resources.
    – Ian Dickinson
    Commented May 17, 2012 at 8:31

1 Answer 1

Reset to default
2

So the question I have, which I can't find an answer to so far, is: if an allowed /id page 303-redirects to a /doc page, will it still be blocked by robots.txt?

From a crawlbot or search engine that obeys robots.txt: Yes.

If I put a link to your.com/id or your.com/doc on my own website, Google will crawl it, follow redirect, read your robots.txt and disallow it from being indexed.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.