I am wondering if a
disallow directive needs to be issued to robots on pages that will generate:
- a 403 forbidden error;
- a blank page, or;
- a page redirect (to log in page for example).
Would not doing so have any impact on SEO of the site?
Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.Sign up to join this community
Will they be indexed:
From the above, you should be able to see that assuming everything is configured correctly, then in cases (1) and (3) we don't need to do anything else to manage these pages' SEO impact.
In the case of (2), we probably don't need to do anything else for a genuinely blank page, though I'd recommend avoiding generating blank pages at all if possible. If not,
Disallow is better than nothing, and if you can apply
"noindex" directly to the page either via HTML or HTTP header, better still.
You should block these directories in the robots, not so much for ranking purposes because it doesn't matter Google often finds lots of 403 and its not irregular for Google to find them. But it'll clutter your Web Master Tools so it is best to block them.
Block Registered User Areas with the Following:
User-agent: * Disallow: /user-area-here-change-me/
Also use noindex as Google recommends both: On all registered area pages use:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">