Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would they ever use them? Thanks

share|improve this question
up vote 3 down vote accepted

Some reasons a bot may be used that come to mind for me are:

  • Some directories will crawl a website for its meta information (usually after a site has been submitted to the directory)

  • Contextual ad services like Adsense use bots to figure out what a page is about so relevant ads can be displayed

  • Webmasters can use bots to find potential linking partners. This may fall into the spammer category but I want to list it for the sake of people who don't use it maliciously.

  • Webmasters can use bots to verify that a link posted in a link exchange has been and still is on the partner's site

share|improve this answer
Good list. One more: Bots are used to scan websites for updates by aggregation services (even if they could use RSS feeds sometimes). – Itai Jul 3 '11 at 1:47
+1 I totally forgot about that. – John Conde Jul 3 '11 at 1:55

You could use a spider to crawl your own site looking for broken links.

share|improve this answer

There are many, many potential uses for web crawlers - a review of Wikipedia's list of Open Source Web Crawlers or examples of Spidering Hacks provided by O'Reilly may help illustrate the numerous ways webmasters and programmers utilize others' services and information.

In addition to "friendly" spiders, you can also expect to see plenty of spambot information harvesters, scrapers, contact form spammers, and vulnerability scanners (like Nessus) in your site's access logs.

Anyone who is familiar with scripting and has an interest in other sites' services can implement a spider of some variety.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.