What benefits would using a web crawler give you as a webmaster?

  • Could you elaborate a little more. Are you referring to a crawler/bot that is attempting to index/log every location that it views with information about each page or something more alone the lines of a scrapping bot to generate content?
    – Analog
    Commented Mar 29, 2016 at 18:09
  • I am referring to a bot that searches through your website and indexes the location and some contents. I do not know what is a scrapping bot that generates content.
    – yoyo_fun
    Commented Mar 29, 2016 at 18:18
  • Are you talking about implementing a site search option??
    – closetnoc
    Commented Mar 29, 2016 at 20:06

2 Answers 2


An internal web crawler can be used for the following purposes:

  1. Creating a Localized Search Engine.
  2. Scanning Pages to Detect Dead Links, where the target page was deleted or unpublished from a CMS, but the links to that page still exist in other pages.
  3. Finding Breadcrumb Click Trail Paths in massive websites.
  4. Finding Orphaned Pages in CMS generated websites.
  5. Load Testing from multiple server locations & different countries.
  6. Unit Testing Pages, which have Jasmine like Green Dots at the bottom of the pages.
  7. Detecting SEO Optimizations on Various Pages, like missing Meta tags.
  8. Generating Customized Reports, which log file analysis tools might not create.
  9. Spell-Checking Pages when working on large sites, where editors create content in a CMS and may not spell-check their articles.
  10. Automatic Foreign Language Page translation (Internationalization AKA "i18n" purposes).
  11. Showing Off Your Programming Smarts to Your I.T. Co-Workers
  12. Anything else that you'd like your bot to test out on massive websites, where database queries might not be as beneficial as an internal web crawler.

You don't have to run a web crawler on your production box. You can run it on a sandbox, like staging or test to see how it would perform under load. Then you could run up 10,000 virtual users & see what the resulting metrics data looks like. You can look at & analyze anything that you want to with any public HTML data, which is on a website.

Most of the time external web crawlers are used by search engines to find content on other people's websites. So unless you're looking to build a search bot & a search engine, you may want to see if any of those internal purposes might be of benefit to you. They can be useful on massive 1,000+ page websites, but won't be beneficial on small 10 page websites. It takes a lot of time to write & debug webcrawlers. So plan wisely, before deciding to build one from scratch!

  • Thank you. This is what I was looking for. Although there are some concepts in your answer that I do not yet understand I will read about them. Do you use a web crawler for your website?
    – yoyo_fun
    Commented Apr 15, 2016 at 11:17
  • Yes. I've built a real external search bot, as I built my own search engine. I crawled 100,000+ website URLs with it. I've discovered that other people will write really messed up querystring parameters & URL paths, which will constantly crash external bots. So I've had to constantly adapt the bot to handle whatever they throw at it. If you make an internal search bot, you will have direct control over how your qs params look. It will be easier to build & require less fine tuning. Building one involves manipulating strings of data to make it do anything that you want it to do. Good luck!
    – Clomp
    Commented Apr 15, 2016 at 16:06
  • that is really cool. I know that there is no limit about the amount of characters in the query string and that most browsers support tens of thousands as character for the GET parameter, so I guess building a web crawler presents some difficulties, considering that people also use special characters to build their queries. Good luck!
    – yoyo_fun
    Commented Apr 15, 2016 at 16:30
  • Yes, those are some of the things that have to be taken into consideration, especially when the database field width is limited or the website is intended to be in English, instead of in a foreign language. Automatically generated ad URLs can mess things up too... but bots are a lot of fun to build! If you build an external one, make sure that it sleeps for 5 seconds per site & identifies itself in the headers, so that webmasters will not flag it as a DDoS attack & will want to white-list it. The bot can still crawl other sites during those 5 seconds, but slow crawl like a person on each site.
    – Clomp
    Commented Apr 15, 2016 at 17:09

Using a crawler on your own site can be good if you get a lot of pages on your site which don't necessarily have an entry in a database to search, this is effectively a way to implement site search for your site. It can also be used where you need a single unified search for a number of separate web properties. I implemented this for a university which had over 100 separate sites under them, some in sub domains some in completely separate domains, all on different servers with different databases. It provided a single unified search tool to search the entire universities online presence easily.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.