Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have A LOT of SHTML pages I want to protect from crawlers, spiders & scrapers.

I understand the limitations of SSIs. An implementation of the following can be suggested in conjunction with any technology/technologies you wish:

The idea is that if you request too many pages too fast you're added to a blacklist for 24 hrs and shown a captcha instead of content, upon every page you request. If you enter the captcha correctly you've removed from the blacklist.
There is a whitelist so GoogleBot, etc. will never get blocked.

Which is the best/easiest way to implement this idea?

Server = IIS

Cleaning out the old tuples from a DB every 24 hrs is easily done so no need to explain that.

share|improve this question
2  
I take it you're annoyed with bots that don't honor robots.txt? And how do you define "you"? By IP address? –  David Thornley May 13 '11 at 19:43
    
@DavidThornley Yes, by IP address –  Adam Lynch May 13 '11 at 20:03
    
The answer might lie in the reason why you are trying to limit the scraping. Is it causing a performance issue? Is there sensitive data being sent? –  Dave Wise May 13 '11 at 20:38
1  
Migrating to Webmasters at Adam's request. –  Anna Lear May 13 '11 at 20:38
    
@DaveWise Sensitive data. Not sensitive as in passwords. This information brings in the money. If we allowed anyone take it; no more money. –  Adam Lynch May 13 '11 at 20:42
add comment

migrated from programmers.stackexchange.com May 13 '11 at 20:39

This question came from our site for professional programmers interested in conceptual questions about software development.

4 Answers

You didn't specify a server technology so this answer may not apply, but can't you simply move the SSI pages to a directory that only the ID that the web server is running as has access to but the anonymous ID does not?

share|improve this answer
    
@DaveWise In the question I say I am running IIS. But I don't want to block typical users –  Adam Lynch May 13 '11 at 20:02
    
Perhaps I misunderstood your question. Are you looking to lock down the SHTML pages themselves, or merely all of the various includes that are used in them? –  Dave Wise May 13 '11 at 20:25
    
I want to protect my SHTML pages from being scraped –  Adam Lynch May 13 '11 at 20:28
    
So, for the sake of your question, the fact that they are SHTML is irrelevant, they could just as easily be plain HTML or ASPX and you would still want them protected, correct? You might want to look into the SPAM/DOS protection that is enabled in some of your network devices (i.e. F5) as they may already have that capability. –  Dave Wise May 13 '11 at 20:35
1  
The answer may be to write your own HttpModule and have it handle this by tracking IP Addresses and the number of requests and then just denying the request if it exceeds your limits. This is pretty simple to do. –  Dave Wise May 13 '11 at 20:53
show 2 more comments

Well, you can render everything into an image format...That sort of thing tends to cause me issues, and it can be done reliably using ImageMagick, for example. I tend to scrape a lot of government sites, and they store vast quantities of information in scanned documents. Blech. What a pain. But judicious use of OCR will foil this sort of security.

If it can be viewed, it can be scraped. Your only recourse is to attempt to identify "mechanical" traffic by monitoring incoming requests, and checking the intervals between page requests. If an ip is requesting multiple pages a second, it's almost certainly a scraper or a spider. Or if it requests one page every 10 seconds, or some other impossibly precise interval. Google uses a learning algorithm to spot scraper-like traffic, but I can count on one hand the number of times I've tripped it (though I very seldom run up against Google content).

A clever scripter will have a random amount of delay built in, however. If they are patient, there is effectively nothing you can do to stop them. Perhaps set an upper limit per IP? You risk alienating your biggest users.

Some people try blocking unknown HTTP_USER_AGENTs, but that's a waste of time: it'll only stop the same people who would respect a robots.txt file.

share|improve this answer
    
The second paragraph is what I want to do. I know what I want to do. It's just I'm not sure how to do it because SHTML/SSIs are limited. –  Adam Lynch May 14 '11 at 14:49
add comment

Are you providing instructions to bots in a robots.txt file?

Are you using a meta element with a "noindex" value in its content attribute?

Have you specified a slower crawl rate in the Google Webmaster (or whichever crawler you are having issues with) interface?

share|improve this answer
    
The type of spiders/crawlers I'm talking about here would not be Google, etc. it would be someone trying to steal our content –  Adam Lynch Jun 7 '11 at 8:09
    
If you want to lock down content you have to use authentication (e.g. username/password). –  Will Peavy Jun 8 '11 at 1:16
    
We have authentication but not in the way you're talking. If we lock down content then that's bad for business. It's proving that you get more sales if you don't force users to register/log in before browsing your products –  Adam Lynch Jun 8 '11 at 8:20
add comment

make a spider trap:

  1. make a page like /spider-trap/somepage.html
  2. block the page on robots.txt: Disallow: /spider-trap/
  3. place a link to this page but hide it to human eyes
  4. block the IP for ANYTHING reach this page
  5. show a human readable hints and an unlock IP captcha something on this page.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.