55 reputation
4
bio website
location
age
visits member for 1 year, 11 months
seen Apr 28 at 15:41

Want to learn everything but takes me a long time.


Jan
11
comment Force fresh content, not from the browser cache
Thanks, I will try that
Oct
5
comment Number of page requests by any Bot in 5 secs
Actually I just pasted the test code on here. I would be using more refined codes for real implementation. I wanted this script only for bots without sessions as I have a number of other scripts that will protect my site for those conditions. Thanks for your help. This has been an educational thread.
Oct
4
comment Number of page requests by any Bot in 5 secs
According to developers.google.com/webmasters/control-crawl-index/docs/… 4xx (client errors) - Google treats all 4xx errors in the same way and assumes that no valid robots.txt file exists. It is assumed that there are no restrictions. This is a "full allow" for crawling. Note: this includes 401 "Unauthorized" and 403 "Forbidden" HTTP result codes.
Oct
3
comment Number of page requests by any Bot in 5 secs
Thanks for taking the time to reply. The request table only has about max 80 rows at any given time. The script above has 800 set so that I want to trigger the condition only if there really is a need for it. I am hosting a site elsewhere and cant really play with the firewall. Thanks anyways.
Oct
3
comment Number of page requests by any Bot in 5 secs
Since I did not get any suggestions, I am setting it to 800 requests in 2 secs. This means my script will block all the bots if there are 800 requests or more in the past 2 secs.