Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed.

Can it be reliably achieved without publishing that personal content in non-document format (e.g. image)?

share|improve this question
    
Do you want to keep the page out of search engines, or just the information on it? Yes, it turns out there's a difference. –  Ilmari Karonen Jan 15 '12 at 14:42
    
If you want the information to remain private (i.e. not available online) do not post it online - scraper bots do not respect robot exclusion protocols. –  danlefree Jan 17 '12 at 14:11
add comment

1 Answer 1

If it's a HTML document, include the following meta tag in it:

<meta name="robots" content="noindex" />

This will tell Google and every other search engine that understands the tag (which should be pretty much all of them, since it's been around since 1996) not to index the page.

For non-HTML documents, you can configure your webserver to send the HTTP header X-Robots-Tag: noindex instead. This is a somewhat newer feature, but it's understood at least by the major search engines like Google, Yahoo! and Bing.

Yet one more way to keep search engines away from a page is to disallow it in your robots.txt file. However, there's an important caveat: Google, at least, treats robots.txt as a prohibition against fetching the disallowed pages, but not against including their URLs in their index if they're linked from another page they've crawled. And you can't effectively combine robots.txt with the meta tag or HTTP header based methods, since disallowing a page in robots.txt will keep Google from ever even seeing any meta tags or HTTP headers on it.

(However, if you disallow a page in your robots.txt and send Google a removal request for it, they'll keep it out of their index. But that's a somewhat laborious and error-prone method.)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.