I have a site with user generated pages where the user can make their page "published" or "unpublished".
My problem is, their pages are often indexed high on Google (usually in the top 3) and so when they unpublish their page, I want their listing removed from Google. I know I can do this via the webmaster tools, but if the user publishes their page again then I would want their page re-listed on Google.
What's the best way to do this?
Simple answer, only slightly difficult solution- robots.txt. It is the fastest and most effective way of removing a page from the index. You can also noindex the page, but that depends upon having Google revisit the page. The robots.txt is much much quicker though nothing is real-time. Google will revisit the robots.txt after 24 hours has expired in conjunction with another crawl which could be a number of days on low activity sites.
You can write code that edits the robots.txt file very easily. When a page is changed from published to unpublished and the page is not new, you can automatically make an entry in the robots.txt file. You can also cause links to this page to be marked as noindex assuming you have some control over this in code. I would age any entry in robots.txt and remove after a certain period of time to make sure that the file does not grow too large. The good news is, you do not have to keep an entry too long. To be safe, you can keep it for a few weeks or months depending upon your activity level.
You, of course, can do this manually too.
