Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Possible Duplicate:
What are the most important things I need to do to encourage Google Sitelinks?

I'm currently working on a website and need to keep certain pages hidden from Google crawler. How to make it so that search engines see only what I want them to see in a directory?

Also, you know how Google results also give you shortcut links, Like 'Login', 'About' etc... how to put these links to search result?

share|improve this question

migrated from stackoverflow.com Nov 23 '11 at 15:45

This question came from our site for professional and enthusiast programmers.

marked as duplicate by John Conde Nov 23 '11 at 15:49

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
The questions pertaining to blocking Google's crawler can be found in many of the questions tagged "robots.txt" –  John Conde Nov 23 '11 at 15:52

4 Answers 4

You can create a robots.txt that disallows those locations.
Beware that not all crawlers will respect robots.txt.

Also, robots.txt is the first place an attacker will look.
You should protext those pages with a secure authentication system.

share|improve this answer

Robots.txt.

share|improve this answer

You can also use <meta name="robots" content="noindex,follow" />

share|improve this answer

If you'd like Google to not list certian URLs on your site, add them to robots.txt in the root of your site.

Example robots.txt:
User-agent: *
Disallow: /donotindex/

Note that this is not secure, you're just "asking" Google to not index/list them. Google will comply with this request. However, sophisticated users and some poorly-behaved crawlers will use or list the URLs regardless.

If you need to hide the content you'll need to put it behind a login page (which crawlers can't use) or change your content based on the client (browser's) user-agent. I believe the latter is specifically disallowed by Google and they may punish your site for doing go.

share|improve this answer