Anyone use Spiders apart from search engines and spammers?

Question

Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would they ever use them? Thanks

Answer 1

Some reasons a bot may be used that come to mind for me are:

• Some directories will crawl a website for its meta information (usually after a site has been submitted to the directory)

• Contextual ad services like Adsense use bots to figure out what a page is about so relevant ads can be displayed

• Webmasters can use bots to find potential linking partners. This may fall into the spammer category but I want to list it for the sake of people who don't use it maliciously.

• Webmasters can use bots to verify that a link posted in a link exchange has been and still is on the partner's site

Answer 2

You could use a spider to crawl your own site looking for broken links.

Answer 3

There are many, many potential uses for web crawlers - a review of Wikipedia's list of Open Source Web Crawlers or examples of Spidering Hacks provided by O'Reilly may help illustrate the numerous ways webmasters and programmers utilize others' services and information.

In addition to "friendly" spiders, you can also expect to see plenty of spambot information harvesters, scrapers, contact form spammers, and vulnerability scanners (like Nessus) in your site's access logs.

Anyone who is familiar with scripting and has an interest in other sites' services can implement a spider of some variety.