1

Possible Duplicate:
What are the most important things I need to do to encourage Google Sitelinks?

I'm currently working on a website and need to keep certain pages hidden from Google crawler. How to make it so that search engines see only what I want them to see in a directory?

Also, you know how Google results also give you shortcut links, Like 'Login', 'About' etc... how to put these links to search result?

Improve this question
1
  • The questions pertaining to blocking Google's crawler can be found in many of the questions tagged "robots.txt"
    – John Conde ♦
    Nov 23, 2011 at 15:52

4 Answers 4

Reset to default
1

You can create a robots.txt that disallows those locations.
Beware that not all crawlers will respect robots.txt.

Also, robots.txt is the first place an attacker will look.
You should protext those pages with a secure authentication system.

Improve this answer
1

You can also use <meta name="robots" content="noindex,follow" />

Improve this answer
1

If you'd like Google to not list certian URLs on your site, add them to robots.txt in the root of your site. 

Example robots.txt:
User-agent: *
Disallow: /donotindex/

Note that this is not secure, you're just "asking" Google to not index/list them. Google will comply with this request. However, sophisticated users and some poorly-behaved crawlers will use or list the URLs regardless.

If you need to hide the content you'll need to put it behind a login page (which crawlers can't use) or change your content based on the client (browser's) user-agent. I believe the latter is specifically disallowed by Google and they may punish your site for doing go.

Improve this answer
1

Robots.txt.

Improve this answer