Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I can see in the stats on my hosting account that there is a big difference between normal page visits and bots visits. I think the bots are causing high cpu usage on the server.
What should I search on in the web access logs to try and identify the problem, and once identified, how can I set up access controls to prevent that?
If by bots, you mean crawlers, then a simple robots.txt file should be able to correct the problem for bots that actually adhere to it. There is little that can be done to prevent those that do not, however, except by blocking them at the network level.
The robots.txt file can be used to whitelist or blacklist crawlers or specific parts of the web site. If you have some static material (or static versions of database-driven copy) that can be served instead, you can point crawlers there without sacrificing too much from a freshness perspective.
