Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

if I want to filter out all the search engine bots for example for a script that counts users that are online is there website or a list which is allways up to date and includes the ip addresses of all the bots? Because if I put them into my script manually I will have to change it every day since there are new ips coming.

Thanks for your help!

share|improve this question

migrated from stackoverflow.com Mar 15 '11 at 16:59

This question came from our site for professional and enthusiast programmers.

You should filter search engines by looking for user agents, not by IP.

share|improve this answer
Unfortunately some server logs do not include user agent information. – JordanReiter Dec 23 '14 at 19:35

From John Conde's reply to a question of mine, see IP Lists. That site lists known IPs used by each search engine.

Filtering by user-agent is not reliable because it can be faked extremely easily. Google suggests looking at the user-agent and then doing a DNS lookup. This way, if they add new IPs, the filtering does not break. You obviously have to update your DNS valiation code when new search engines or bots appear by that would not be too often.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.