Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I can see in the stats on my hosting account that there is a big difference between normal page visits and bots visits. I think the bots are causing high cpu usage on the server.

What should I search on in the web access logs to try and identify the problem, and once identified, how can I set up access controls to prevent that?

share|improve this question
add comment

migrated from security.stackexchange.com Feb 19 '12 at 17:34

This question came from our site for Information security professionals.

1 Answer

If by bots, you mean crawlers, then a simple robots.txt file should be able to correct the problem for bots that actually adhere to it. There is little that can be done to prevent those that do not, however, except by blocking them at the network level.

The robots.txt file can be used to whitelist or blacklist crawlers or specific parts of the web site. If you have some static material (or static versions of database-driven copy) that can be served instead, you can point crawlers there without sacrificing too much from a freshness perspective.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.