3

I'm trying to follow stackoverflow's approach of using a white list for the sitemap. I have tried to find an extensive list of search engine crawlers (and other crawlers) to use in my white list but what I have found so far is outdated information (hasn't been updated in years). Is there any place on the net where I can find a regularly updated list that contains the user agent of every search engine crawler along with its domain name (for reverse DNS)?

Thanks

Improve this question

3 Answers 3

Reset to default
1

Here's how to verify Googlebot: Verifying Googlebot

This seems to cover BING indirectly. A script that may help you narrow down when Bing is on your site (in PHP).

Improve this answer
1
  • Thanks. I think I'll do it this way (look for the details for every bot individually). I don't intend to whitelist too many bots, only the major ones, so it's no big deal. I was just hoping to save some time by using an already updated list, but unfortunately such list doesn't seem to exist.
    – Dev
    Sep 25, 2010 at 2:29
1

The question you referenced indicates that the answerer is whitelisting the user agents themselves, not the domains or IP addresses associated with particular spiders.

It appears as though user-agents.org has an exhaustive list if you plan to work with the user-agent string.

Improve this answer
4
  • Whitelisting the user agents isn't enough as they can easily be spoofed (see meta.stackexchange.com/questions/37231/…). I haven't mentioned anything about IP addresses but I need the domain name for the reverse DNS lookup.
    – Dev
    Sep 25, 2010 at 2:23
  • The next logical step (after whitelisting user agents and rDNS info*) would be to white list IP addresses if you're concerned about "unauthorized" traffic on your sitemap... but the information on your sitemap is intended to become public in some form sooner or later. Have you considered that locking up your sitemap is more trouble than it is worth? * - Note: rDNS records can also be spoofed, if that bears mentioning.
    – danlefree
    Sep 25, 2010 at 4:00
  • Dan, I agree that the file should be public, unless it's being abused. Stackoverflow does this for good reasons as the file was being downloaded by non-search-engine-spiders and that was consuming too much bandwidth. Whitelisting the IP addresses is not actually a good idea as they can change. It's true that rDNS alone is not enough, this is why you should do a forward DNS lookup next as part of the check (Please see google.com/support/webmasters/bin/answer.py?answer=80553)
    – Dev
    Sep 25, 2010 at 15:25
  • Tangential: If it's an abuse situation I might actually recommend whitelisting the IP addresses you've identified as legit (and blacklisting the IP addresses you've identified as illegitimate) to speed things along - once you've authenticated a user agent and associated forward and reverse DNS records it's pretty safe to assume abusive activity won't be masquerading under identical credentials (and vice versa)... but if abuse isn't presently occurring, it's a lot of additional work for marginal benefit (i.e. the doc you linked ends with the advice to just use the user agent string).
    – danlefree
    Sep 25, 2010 at 22:38
0

Maybe a better approach than trying to whitelist every search engine is to go for the big three (Google, Yahoo, Bing) plus possibly Ask (which are known to support sitemaps, too).

Whitelisting all of them seems to be a long, hard and in the end futile task. You can still log who's banging on the door and expand your list based on that.

Improve this answer
1
  • Actually I wasn't planning to whitelist all of them but I was hoping there's a list with all the required information so that I can pick up the ones I want to whitelist.
    – Dev
    Sep 25, 2010 at 2:26