Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am wondering if a disallow directive needs to be issued to robots on pages that will generate:

  1. a 403 forbidden error;
  2. a blank page, or;
  3. a page redirect (to log in page for example).

Would not doing so have any impact on SEO of the site?

share|improve this question
    
What sort of problems are you anticipating? Google is picky about which pages they index, and even pickier about which pages they return in search results. The types of "pages" (a redirect isn't really a page) you list pretty much have no content, so even if they were indexed, would never get returned in any search results. I also can't think of any reason why a search engine would make the crawlability of these pages a ranking factor. –  Lèse majesté Apr 20 '12 at 16:27
    
@Lèse majesté, I want to know whether the robots will look for anything besides the links that appear on my web pages and whether they can see the links that only appear for registered users. Because if they can, then many pages will come up with forbidden errors. Not sure if too many of these errors would have an overall drag on my website ranking. And if it does, then I would need to explicitly disallow them on robots.txt. –  Question Overflow Apr 21 '12 at 5:58

2 Answers 2

up vote 1 down vote accepted

Will they be indexed:

  1. 403 Error – no, most search engines don't index error pages (assuming they're served with the correct HTTP status code)
  2. Blank page – almost certainly not, assuming it is entirely blank (no <title>, nothing).
  3. Page redirect – Generally only the destination of the redirect will be indexed (usually this is the intended result), but it depends on the type of redirect. For 301 the redirecting page will be de-indexed in favour of the destination page, however for 302 often both the referring and destination page can continue to be indexed.

SEO impact

From the above, you should be able to see that assuming everything is configured correctly, then in cases (1) and (3) we don't need to do anything else to manage these pages' SEO impact.

In the case of (2), we probably don't need to do anything else for a genuinely blank page, though I'd recommend avoiding generating blank pages at all if possible. If not, Disallow is better than nothing, and if you can apply "noindex" directly to the page either via HTML or HTTP header, better still.

share|improve this answer
    
Thanks, I posted this quite a while ago. Had a better understanding since then. I fully agree with what you wrote :) –  Question Overflow Mar 20 '13 at 4:53

You should block these directories in the robots, not so much for ranking purposes because it doesn't matter Google often finds lots of 403 and its not irregular for Google to find them. But it'll clutter your Web Master Tools so it is best to block them.

Block Registered User Areas with the Following:

Robots.txt

User-agent: *
Disallow: /user-area-here-change-me/

Also use noindex as Google recommends both: On all registered area pages use:

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.