I am currently running a small affiliate-program for Facebook users. We use an auto-poster to publish links to fan pages. Every hit is stored in our database and we have included a 24 hour reload block for the IP-addresses. My problem right now is that the PHP script also stores every hit from all the bots that crawls my website. Now I was thinking to block those bots with the robots.txt of my website but I am afraid that this will have a negative effect on my AdSense ads. Does anybody have an idea for me how to work this out?
Avoid using the robots.txt to block known robots. You want robots to crawl your site, you only need to distinguish which hits must be logged and which not.
To solve this problem, you could check the client's user agent and store only the hits that meet your criteria. For example you could use a simple function, to check if the hit comes from a known search engine (You can also add more keys):
And call it with the client's user_agent, retrieved from the