Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed.
Can it be reliably achieved without publishing that personal content in non-document format (e.g. image)?
If it's a HTML document, include the following meta tag in it:
<meta name="robots" content="noindex" />
This will tell Google and every other search engine that understands the tag (which should be pretty much all of them, since it's been around since 1996) not to index the page.
For non-HTML documents, you can configure your webserver to send the HTTP header X-Robots-Tag: noindex instead. This is a somewhat newer feature, but it's understood at least by the major search engines like Google, Yahoo! and Bing.
Yet one more way to keep search engines away from a page is to disallow it in your robots.txt file. However, there's an important caveat: Google, at least, treats robots.txt as a prohibition against fetching the disallowed pages, but not against including their URLs in their index if they're linked from another page they've crawled. And you can't effectively combine robots.txt with the meta tag or HTTP header based methods, since disallowing a page in robots.txt will keep Google from ever even seeing any meta tags or HTTP headers on it.
(However, if you disallow a page in your robots.txt and send Google a removal request for it, they'll keep it out of their index. But that's a somewhat laborious and error-prone method.)
