I am wondering if a disallow directive needs to be issued to robots on pages that will generate:

  1. a 403 forbidden error;
  2. a blank page, or;
  3. a page redirect (to log in page for example).

Would not doing so have any impact on SEO of the site?

  • What sort of problems are you anticipating? Google is picky about which pages they index, and even pickier about which pages they return in search results. The types of "pages" (a redirect isn't really a page) you list pretty much have no content, so even if they were indexed, would never get returned in any search results. I also can't think of any reason why a search engine would make the crawlability of these pages a ranking factor. – Lèse majesté Apr 20 '12 at 16:27
  • @Lèse majesté, I want to know whether the robots will look for anything besides the links that appear on my web pages and whether they can see the links that only appear for registered users. Because if they can, then many pages will come up with forbidden errors. Not sure if too many of these errors would have an overall drag on my website ranking. And if it does, then I would need to explicitly disallow them on robots.txt. – Question Overflow Apr 21 '12 at 5:58

Will they be indexed:

  1. 403 Error – no, most search engines don't index error pages (assuming they're served with the correct HTTP status code)
  2. Blank page – almost certainly not, assuming it is entirely blank (no <title>, nothing).
  3. Page redirect – Generally only the destination of the redirect will be indexed (usually this is the intended result), but it depends on the type of redirect. For 301 the redirecting page will be de-indexed in favour of the destination page, however for 302 often both the referring and destination page can continue to be indexed.

SEO impact

From the above, you should be able to see that assuming everything is configured correctly, then in cases (1) and (3) we don't need to do anything else to manage these pages' SEO impact.

In the case of (2), we probably don't need to do anything else for a genuinely blank page, though I'd recommend avoiding generating blank pages at all if possible. If not, Disallow is better than nothing, and if you can apply "noindex" directly to the page either via HTML or HTTP header, better still.

  • Thanks, I posted this quite a while ago. Had a better understanding since then. I fully agree with what you wrote :) – Question Overflow Mar 20 '13 at 4:53

You should block these directories in the robots, not so much for ranking purposes because it doesn't matter Google often finds lots of 403 and its not irregular for Google to find them. But it'll clutter your Web Master Tools so it is best to block them.

Block Registered User Areas with the Following:


User-agent: *
Disallow: /user-area-here-change-me/

Also use noindex as Google recommends both: On all registered area pages use:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.