I have A LOT of SHTML pages I want to protect from crawlers, spiders & scrapers.

I understand the limitations of SSIs. An implementation of the following can be suggested in conjunction with any technology/technologies you wish:

The idea is that if you request too many pages too fast you're added to a blacklist for 24 hrs and shown a captcha instead of content, upon every page you request. If you enter the captcha correctly you've removed from the blacklist.
There is a whitelist so GoogleBot, etc. will never get blocked.

Which is the best/easiest way to implement this idea?

Server = IIS

Cleaning out the old tuples from a DB every 24 hrs is easily done so no need to explain that.

  • 2
    I take it you're annoyed with bots that don't honor robots.txt? And how do you define "you"? By IP address?
    – David Thornley
    Commented May 13, 2011 at 19:43
  • @DavidThornley Yes, by IP address
    – Adam Lynch
    Commented May 13, 2011 at 20:03
  • The answer might lie in the reason why you are trying to limit the scraping. Is it causing a performance issue? Is there sensitive data being sent?
    – Dave Wise
    Commented May 13, 2011 at 20:38
  • 1
    Migrating to Webmasters at Adam's request.
    – Anna Lear
    Commented May 13, 2011 at 20:38
  • @DaveWise Sensitive data. Not sensitive as in passwords. This information brings in the money. If we allowed anyone take it; no more money.
    – Adam Lynch
    Commented May 13, 2011 at 20:42

4 Answers 4


make a spider trap:

  1. make a page like /spider-trap/somepage.html
  2. block the page on robots.txt: Disallow: /spider-trap/
  3. place a link to this page but hide it to human eyes
  4. block the IP for ANYTHING reach this page
  5. show a human readable hints and an unlock IP captcha something on this page.

You didn't specify a server technology so this answer may not apply, but can't you simply move the SSI pages to a directory that only the ID that the web server is running as has access to but the anonymous ID does not?

  • @DaveWise In the question I say I am running IIS. But I don't want to block typical users
    – Adam Lynch
    Commented May 13, 2011 at 20:02
  • Perhaps I misunderstood your question. Are you looking to lock down the SHTML pages themselves, or merely all of the various includes that are used in them?
    – Dave Wise
    Commented May 13, 2011 at 20:25
  • I want to protect my SHTML pages from being scraped
    – Adam Lynch
    Commented May 13, 2011 at 20:28
  • So, for the sake of your question, the fact that they are SHTML is irrelevant, they could just as easily be plain HTML or ASPX and you would still want them protected, correct? You might want to look into the SPAM/DOS protection that is enabled in some of your network devices (i.e. F5) as they may already have that capability.
    – Dave Wise
    Commented May 13, 2011 at 20:35
  • 1
    The answer may be to write your own HttpModule and have it handle this by tracking IP Addresses and the number of requests and then just denying the request if it exceeds your limits. This is pretty simple to do.
    – Dave Wise
    Commented May 13, 2011 at 20:53

Well, you can render everything into an image format...That sort of thing tends to cause me issues, and it can be done reliably using ImageMagick, for example. I tend to scrape a lot of government sites, and they store vast quantities of information in scanned documents. Blech. What a pain. But judicious use of OCR will foil this sort of security.

If it can be viewed, it can be scraped. Your only recourse is to attempt to identify "mechanical" traffic by monitoring incoming requests, and checking the intervals between page requests. If an ip is requesting multiple pages a second, it's almost certainly a scraper or a spider. Or if it requests one page every 10 seconds, or some other impossibly precise interval. Google uses a learning algorithm to spot scraper-like traffic, but I can count on one hand the number of times I've tripped it (though I very seldom run up against Google content).

A clever scripter will have a random amount of delay built in, however. If they are patient, there is effectively nothing you can do to stop them. Perhaps set an upper limit per IP? You risk alienating your biggest users.

Some people try blocking unknown HTTP_USER_AGENTs, but that's a waste of time: it'll only stop the same people who would respect a robots.txt file.

  • The second paragraph is what I want to do. I know what I want to do. It's just I'm not sure how to do it because SHTML/SSIs are limited.
    – Adam Lynch
    Commented May 14, 2011 at 14:49

Are you providing instructions to bots in a robots.txt file?

Are you using a meta element with a "noindex" value in its content attribute?

Have you specified a slower crawl rate in the Google Webmaster (or whichever crawler you are having issues with) interface?

  • The type of spiders/crawlers I'm talking about here would not be Google, etc. it would be someone trying to steal our content
    – Adam Lynch
    Commented Jun 7, 2011 at 8:09
  • If you want to lock down content you have to use authentication (e.g. username/password).
    – Will
    Commented Jun 8, 2011 at 1:16
  • We have authentication but not in the way you're talking. If we lock down content then that's bad for business. It's proving that you get more sales if you don't force users to register/log in before browsing your products
    – Adam Lynch
    Commented Jun 8, 2011 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.