Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get redirected to dynamically-created pages under /doc. These dynamic pages are built from a database, and contain links to other /doc/ resources, so in general we don't want them to be crawled. Our robots.txt contains:

Disallow: /doc

However, we do want the non-redirected pages under /id to get indexed by Google et al:

Allow: /id

So the question I have, which I can't find an answer to so far, is: if an allowed /id page 303-redirects to a /doc page, will it still be blocked by robots.txt?

If yes, we're OK, but otherwise I'm going to disallow all /id resources in the robots file, as having the crawler hammer the db would be worse than losing search indexing for the /id pages.

share|improve this question
    
Is it possible to make the database-generated pages under /doc/ be put at /id/ rather? That way, you can index what you want, and block what you don't want public. I don't see 303 as a good idea, though: sharkseo.com/nohat/303-redirects-seo –  ionFish May 17 '12 at 1:38
    
The whole point of separating /doc and /id is so that they are different. This allows you to make the distinction between information about a thing and a reference to the thing itself. As for user agents not recognising 303 redirects, that hasn't proved to be a problem in practice, and a primary aim anyway is to support machine processing of semantic web resources. –  Ian Dickinson May 17 '12 at 8:31
add comment

1 Answer 1

So the question I have, which I can't find an answer to so far, is: if an allowed /id page 303-redirects to a /doc page, will it still be blocked by robots.txt?

From a crawlbot or search engine that obeys robots.txt: Yes.

If I put a link to your.com/id or your.com/doc on my own website, Google will crawl it, follow redirect, read your robots.txt and disallow it from being indexed.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.