I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, yahoo, bot, crawler, etc.

Does this mean that those users were infact search engine crawlers?

If so, why are their so many crawlers in my visitor records in comparison to genuine human visitors? (and if not what's the explanation?)

Thanks in advance.


Here's a few examples of the data:








  • 1
    IP addresses are just numbers, and can't contain keywords. Do you mean that the keywords are in reverse DNS lookups of the IP addresses, in the User Agent strings, or somewhere else?
    – Mike Scott
    Jan 11, 2011 at 15:58
  • I think I'm referring to the host name. I have posted an example in the question. So I suppose my question is what does the host name tell me?
    – Roscoe
    Jan 11, 2011 at 16:20
  • Google provides DNS services now, so it could be one of those users.
    – dkuntz2
    Jan 12, 2011 at 6:39
  • @DKuntz2: Using Google's DNS service should not cause Google servers to show up in anyone's web logs. The DNS server has no need to communicate directly with the web server. Jan 12, 2011 at 10:58
  • 1
    They'd show up under the host name for my analytics (so instead of comcast it would say google if they're using google's dns).
    – dkuntz2
    Jan 12, 2011 at 13:50

2 Answers 2


That looks genuine, as it's fairly hard to spoof a source domain name -- crawlers can generate a lot of traffic. They often check pages for changes quite frequently.

You can slow down many of them with a non-standard (but fairly well supported) addition to the Robots Exclusion Protocol -- create a file called robots.txt that's served from your web server's root directory with the following contents:

User-agent: *
Crawl-delay: 60

Where the number on the second line is the number of seconds you want each crawler to wait between page loads on your site. (If you've already got a robots.txt file you'll need to modify it instead.)

  • Thanks mike. Can I just confirm what exactly the data is I'm looking at if it isn't an IP address?
    – Roscoe
    Jan 11, 2011 at 16:31
  • 2
    @Roscoe - You are looking at PTR records. Note that they're easy to forge. Related reading: en.wikipedia.org/wiki/Reverse_DNS_lookup
    – danlefree
    Jan 11, 2011 at 16:37
  • I'm not sure what your example is. I'd say it was a reverse DNS lookup for the IP address of a crawler, but the IP address is presumably (based on the domain name), and the reverse lookup for that is msnbot-65-55-209-133.search.msn.com. Where exactly are you getting it from? Can you post a few complete lines of your logfiles?
    – Mike Scott
    Jan 11, 2011 at 16:41
  • Thanks for the link Dan. I've added some extra examples in the main question. The data is something I have been given so not recorded by myself. I was looking to find out if lines such as those mean that the user was a crawler, or whether it means they came from a search engine.
    – Roscoe
    Jan 11, 2011 at 16:46
  • They mean the user is a crawler, or faking their domain name. The former is much more likely (except that cuil.com has now closed down, and was spelled with one 'l' when it was live -- is this by any chance old data that you've retyped here (with the odd typo) rather than copying and pasting?).
    – Mike Scott
    Jan 11, 2011 at 16:51

These are more commonly referred to as crawler user-agents. The subdomains of the agents normally refer to a server ID (sometimes including the IP address).

You can find a comprehensive list at http://www.user-agents.org/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.