I have seen several newspaper sites that have started to restrict the number of articles the user is allowed to read, before being prompted to register/subscribe.
I know how to programatically restrict the number of articles a user is allowed to read, but I do not want to block Googlebot or other search engines from being able to index all my articles.
It seems like some forum software has a way where Googlebot has their own login account and are able to index all the pages still. I'm not sure just detected the UA would be sufficient as someone could easily spoof that.
Is there a recommended way to do this?