Is there a way to tell search engines, that a page they crawl should be included in the search results now, but have to be deleted at a certain time in the future?
I have a website where hundreds of publications happen each day and I want them to be crawled and be searchable, but I am legally required to remove the information after a while (individual date for each page).
After that given date, the page will not be visible at my website anymore (HTTP response 410 gone), but the page will linger in e.g. the google cache for a while, which could cause legal issues for me. Obviously, it's not viable to issue hundreds of content removal requests to google by hand. On the other hand, the individual pages do not get modified for some months until they have to be discarded, so google bot won't check in often.
For what I understand, the HTTP Expires header is a label for minimum freshnes and not for maximum lifetime, correct? I am sending last-modified-at and etag headers, but they don't help here. Is there any way to say "cache, but only until 2011-08-15"?