Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Is there a way to tell search engines, that a page they crawl should be included in the search results now, but have to be deleted at a certain time in the future?

I have a website where hundreds of publications happen each day and I want them to be crawled and be searchable, but I am legally required to remove the information after a while (individual date for each page).

After that given date, the page will not be visible at my website anymore (HTTP response 410 gone), but the page will linger in e.g. the google cache for a while, which could cause legal issues for me. Obviously, it's not viable to issue hundreds of content removal requests to google by hand. On the other hand, the individual pages do not get modified for some months until they have to be discarded, so google bot won't check in often.

For what I understand, the HTTP Expires header is a label for minimum freshnes and not for maximum lifetime, correct? I am sending last-modified-at and etag headers, but they don't help here. Is there any way to say "cache, but only until 2011-08-15"?

share|improve this question

migrated from stackoverflow.com May 22 '11 at 10:46

This question came from our site for professional and enthusiast programmers.

    
For google you could try: <meta name="googlebot" content="nosnippet"> –  zaf May 20 '11 at 10:31
    
Additionally, the noarchive tag tells Google not to store a cached copy of your page. –  Kris May 20 '11 at 10:32

2 Answers 2

up vote 4 down vote accepted

For google there is a meta tag called unavailable_after, which does exactly what I was looking for: It tells google to remove a certain page at a specific time in the future.

It is the only way to achieve what I was hoping to accomplish: Getting the pages removed automatically, at the right time, not relying on the crawler to come back and notice the 410 Gone response, which can take a weeks after the content has been removed.

Example:

<META NAME="GOOGLEBOT" CONTENT="unavailable_after: 25-Aug-2007 15:00:00 EST" />

Or with HTTP header, for PDFs etc.:

X-Robots-Tag: unavailable_after: 23 Jul 2007 15:00:00 PST

Sources: http://googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-with-even.html and http://www.google.com/support/webmasters/bin/answer.py?answer=79812

I could not find out if Bing, Yahoo & Co have adopted this Google specific tag.

share|improve this answer

First of all you don't have control what search engines crawl and what they put in their index.

BUT, Google for instance is taking your information about the live time of your pages very serious. So if you add the correct HTTP header it will consider those information. You can also add some information into your robots.txt about which pages are invalid.

There are also the Webmaster tools where you can tell Google to remove pages from the index.

On the official Google webmaster blog you will find very helpful information about removing URLs from the index and how to reinclude content. There they say you can remove URLs by:

  • using 410,
  • robots.txt or the
  • noindex meta tag
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.