2

if I want to filter out all the search engine bots for example for a script that counts users that are online is there website or a list which is allways up to date and includes the ip addresses of all the bots? Because if I put them into my script manually I will have to change it every day since there are new ips coming.

Thanks for your help!

Improve this question

2 Answers 2

Reset to default
3

You should filter search engines by looking for user agents, not by IP.

Improve this answer
1
  • Unfortunately some server logs do not include user agent information.
    – JordanReiter
    Dec 23, 2014 at 19:35
3

From John Conde's reply to a question of mine, see IP Lists. That site lists known IPs used by each search engine.

Filtering by user-agent is not reliable because it can be faked extremely easily. Google suggests looking at the user-agent and then doing a DNS lookup. This way, if they add new IPs, the filtering does not break. You obviously have to update your DNS valiation code when new search engines or bots appear by that would not be too often.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.