Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11

I am thinking of adding the following line to my robots.txt file:

Disallow: /some/fixed/path/admin/*
share|improve this question
add comment

3 Answers

up vote 13 down vote accepted

There are 2 main ways to prevent search engines from indexing specific pages:

  1. A Robots.txt file for your domain.
  2. The Meta Robots tag on each page.

Robots.txt should be your first stop for URL patterns that match several files. You can see the syntax here and more detailed here. The robots.txt file must be placed in the root folder of your domain, i.e. at http://www.yourdomain.com/robots.txt , and it would contain something like:

User-agent: *
Disallow: /path/with-trailing-slash/

(The text coloring above is done by the Stackexchange software, and should be ignored.)

The Meta Robots tag is more flexible and capable, but must be inserted in every page you want to affect.

Again Google has a overview of how to use Meta Robots, and how to get pages removed from their index via Webmaster Tools. Wikipedia has more comprehensive documentation on Meta Robots, including the search engine specific derivations.

If you want to prohibit Google, The Web Archive and other search engines from keeping a copy of your webpage, then you want the following tag (shown in HTML4 format):

<meta name="robots" content="noarchive">

To prevent indexing and keeping a copy:

<meta name="robots" content="noindex, noarchive">

And to prevent both of the above, as well as using links on the page to find more pages to index:

<meta name="robots" content="noindex, nofollow, noarchive">

NB 1: All 3 above meta tags are for search engines alone -- they do not impact HTTP proxies or browsers.

NB 2: If you already have pages indexed and archived, and you block pages via robots.txt while at the same time adding the meta tag to the same pages, then the robots.txt will prevent search engines from seeing the updated meta tag.

share|improve this answer
1  
Downvoted? Why on earth was this downvoted? Please leave a comment if you down-vote so the answer can be improved. –  Jesper Mortensen Sep 11 '10 at 12:16
    
@Jesper Mortensen Your initial answer did not address the caching question at all. Your edit fixed this and made the noindex info much better. +1 now ;-) –  mawtex Sep 11 '10 at 12:46
1  
One thing to keep in mind is that a robots.txt disallow directive does not prevent indexing of a URL nor does it result in the removal of that URL from the index. Search engines can and will index URLs without having crawled them (if they're disallowed), so if stopping URLs from being indexed is critical (and not just stopping content from being indexed), then you must use the robots meta tag or the x-robots-tag HTTP header and make sure that the URLs are not disallowed from crawling. –  John Mueller Sep 12 '10 at 14:09
1  
Also, while not necessarily incorrect, a robots meta tag with "noindex, noarchive" is equivalent to "noindex" (when a URL is not indexed, it's not archived/cached either). –  John Mueller Sep 12 '10 at 14:13
1  
Finally (sorry for adding so many comments :-)), in this particular case (admin-pages), I would just make sure that the URLs return 403 when not logged in. That also prevents search engines from indexing it and is theoretically clearer than having a page return 200 + using a noindex robots meta tag. The end result is the same in the search results, but using the proper HTTP result code can help you to recognize unauthorized admin-accesses in your logs easier. –  John Mueller Sep 12 '10 at 14:18
show 1 more comment

There's actually a third way to prevent Google and other search engines from indexing URLs. It's the X-Robots_tag header. This is better then meta tags because it works for all documents and you can have more then one tag.

The REP META tags give you useful control over how each webpage on your site is indexed. But it only works for HTML pages. How can you control access to other types of documents, such as Adobe PDF files, video and audio files and other types? Well, now the same flexibility for specifying per-URL tags is available for all other files type.

We've extended our support for META tags so they can now be associated with any file. Simply add any supported META tag to a new X-Robots-Tag directive in the HTTP Header used to serve the file. Here are some illustrative examples: Don't display a cache link or snippet for this item in the Google search results: X-Robots-Tag: noarchive, nosnippet Don't include this document in the Google search results: X-Robots-Tag: noindex Tell us that a document will be unavailable after 7th July 2007, 4:30pm GMT: X-Robots-Tag: unavailable_after: 7 Jul 2007 16:30:00 GMT

You can combine multiple directives in the same document. For example: Do not show a cached link for this document, and remove it from the index after 23rd July 2007, 3pm PST: X-Robots-Tag: noarchive X-Robots-Tag: unavailable_after: 23 Jul 2007 15:00:00 PST

share|improve this answer
    
The 'X-Robots_tag header' link is broken. –  mawtex Sep 11 '10 at 15:16
    
Thanks for the heads up. Chrome seems to have issues with the formatting toolbar and it added extra text to the link. –  John Conde Sep 11 '10 at 16:09
add comment

Yes, that will fix the problem. To prevent content from showing up in Googles indexes you can either use robots.txt or the html meta tag

<meta name="robots" content="noindex, nofollow" />

The next time your site is indexed this will make your content drop out of the Google index.

You can also you the noarchive value – this will block caching of your page. This is Google specific:

<meta name="robots" content="noarchive" />

You can use the ‘removal tool’ in Googles Webmaster Tools to request a very urgent removal of your content. Note that you should block indexing of your content first (using either robots.txt or the meta robots tag).

More info:

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.