Are spiders ever used by anyone other than search engines and spammers? If someone wanted to collect information from the web, would they ever use them? Thanks


Some reasons a bot may be used that come to mind for me are:

  • Some directories will crawl a website for its meta information (usually after a site has been submitted to the directory)

  • Contextual ad services like Adsense use bots to figure out what a page is about so relevant ads can be displayed

  • Webmasters can use bots to find potential linking partners. This may fall into the spammer category but I want to list it for the sake of people who don't use it maliciously.

  • Webmasters can use bots to verify that a link posted in a link exchange has been and still is on the partner's site

  • 2
    Good list. One more: Bots are used to scan websites for updates by aggregation services (even if they could use RSS feeds sometimes).
    – Itai
    Jul 3 '11 at 1:47

You could use a spider to crawl your own site looking for broken links.


There are many, many potential uses for web crawlers - a review of Wikipedia's list of Open Source Web Crawlers or examples of Spidering Hacks provided by O'Reilly may help illustrate the numerous ways webmasters and programmers utilize others' services and information.

In addition to "friendly" spiders, you can also expect to see plenty of spambot information harvesters, scrapers, contact form spammers, and vulnerability scanners (like Nessus) in your site's access logs.

Anyone who is familiar with scripting and has an interest in other sites' services can implement a spider of some variety.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.