I have seen several newspaper sites that have started to restrict the number of articles the user is allowed to read, before being prompted to register/subscribe.

e.g. http://www.heraldnet.com/

I know how to programatically restrict the number of articles a user is allowed to read, but I do not want to block Googlebot or other search engines from being able to index all my articles.

It seems like some forum software has a way where Googlebot has their own login account and are able to index all the pages still. I'm not sure just detected the UA would be sufficient as someone could easily spoof that.

Is there a recommended way to do this?


I'd suggest using the google crawler user agent and add a condition to your logic... Here's a list of user agents used by the G Crawlers.. support.google. com/webmasters/answer/1061943?hl=en In fact there's another way to give access to the spider with "special user account" i'd rather suggest accessing your google webmaster tools, in the "authorization" section where you have to precise where it should send the post request and the data etc...

EDIT 1 : Reverse lookup to safely check if its actually GoogleCrawler

EDIT 2 : Google WebMaster Tools URL parameters

The use of the URL parameters : GoogleCrawler will try to access your website with another _GET parameters for example if you set the parameters as follows (user=google,pwd=crawler009) then if you had yoursite.com/login.php , The crawler will access your website with yoursite.com/login.php?user=google&pwd=crawler009

What you have to do is to add a sort of "logic" to accept logins even from the GET params or simply make an exception for the crawler, its almost the same we do with _POST params :)

| improve this answer | |
  • "using the google crawler user agent" - you will also need to validate the IP address of the crawler, otherwise it would be trivial for anyone else to gain access by simply faking their UA. Can you provide information about this "special user account"? And where is the "authorization" section in Google Search Console (formerly GWT)? – MrWhite Nov 30 '16 at 9:35
  • Thanks @w3dk, i forgot to mention the ip thingy... in fact since the crawler's ips change often, google suggests using a reverse DNS lookup to check whenever its the GoogleCrawler or not, here's an article from google support. – Yidir Dec 6 '16 at 2:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.