Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 0 down vote favorite

I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, yahoo, bot, crawler, etc.

Does this mean that those users were infact search engine crawlers?

If so, why are their so many crawlers in my visitor records in comparison to genuine human visitors? (and if not what's the explanation?)

Thanks in advance.

UPDATE

Here's a few examples of the data:

livebot-65-55-209-133.search.live.com

crawl2.cosmixcorp.com

crawl-66-249-70-78.googlebot.com

lj511965.crawl.yahoo.net

lj611054.inktomisearch.com

ss125.dal0.gigablast.com

crawl-15.cuill.com

share|improve this question
1  
IP addresses are just numbers, and can't contain keywords. Do you mean that the keywords are in reverse DNS lookups of the IP addresses, in the User Agent strings, or somewhere else? – Mike Scott Jan 11 '11 at 15:58
I think I'm referring to the host name. I have posted an example in the question. So I suppose my question is what does the host name tell me? – Roscoe Jan 11 '11 at 16:20
Google provides DNS services now, so it could be one of those users. – dkuntz2 Jan 12 '11 at 6:39
@DKuntz2: Using Google's DNS service should not cause Google servers to show up in anyone's web logs. The DNS server has no need to communicate directly with the web server. – Lèse majesté Jan 12 '11 at 10:58
1  
They'd show up under the host name for my analytics (so instead of comcast it would say google if they're using google's dns). – dkuntz2 Jan 12 '11 at 13:50
show 1 more comment

2 Answers

up vote 2 down vote accepted

That looks genuine, as it's fairly hard to spoof a source domain name -- crawlers can generate a lot of traffic. They often check pages for changes quite frequently.

You can slow down many of them with a non-standard (but fairly well supported) addition to the Robots Exclusion Protocol -- create a file called robots.txt that's served from your web server's root directory with the following contents:

User-agent: *
Crawl-delay: 60

Where the number on the second line is the number of seconds you want each crawler to wait between page loads on your site. (If you've already got a robots.txt file you'll need to modify it instead.)

share|improve this answer
Thanks mike. Can I just confirm what exactly the data is I'm looking at if it isn't an IP address? – Roscoe Jan 11 '11 at 16:31
2  
@Roscoe - You are looking at PTR records. Note that they're easy to forge. Related reading: en.wikipedia.org/wiki/Reverse_DNS_lookup – danlefree♦ Jan 11 '11 at 16:37
I'm not sure what your example is. I'd say it was a reverse DNS lookup for the IP address of a crawler, but the IP address is presumably 65.55.209.133 (based on the domain name), and the reverse lookup for that is msnbot-65-55-209-133.search.msn.com. Where exactly are you getting it from? Can you post a few complete lines of your logfiles? – Mike Scott Jan 11 '11 at 16:41
Thanks for the link Dan. I've added some extra examples in the main question. The data is something I have been given so not recorded by myself. I was looking to find out if lines such as those mean that the user was a crawler, or whether it means they came from a search engine. – Roscoe Jan 11 '11 at 16:46
They mean the user is a crawler, or faking their domain name. The former is much more likely (except that cuil.com has now closed down, and was spelled with one 'l' when it was live -- is this by any chance old data that you've retyped here (with the odd typo) rather than copying and pasting?). – Mike Scott Jan 11 '11 at 16:51
show 2 more comments
up vote 0 down vote

These are more commonly referred to as crawler user-agents. The subdomains of the agents normally refer to a server ID (sometimes including the IP address).

You can find a comprehensive list at http://www.user-agents.org/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.