Possible Duplicate:
How to protect SHTML pages from crawlers/spiders/scrapers?

My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling for that name produces various results of people who've concluded that it doesn't respect robots.txt (eg, http://www.0sw.com/archives/96).

I'm considering updating my app to have a list of banned user-agents, and serving all requests from those user-agents a 400 or similar and adding GSLFBot to that list. Is that an effective technique, and if not what should I do instead?

(As a side note, it seems weird to have an abusive scraper with a distinctive user-agent.)

  • 2
    The only thing it would take to bypass your restriction would be a change in the user-agent string of the bot. – Sherwin Flight Apr 4 '12 at 6:46
  • True, but on the other hand it can be interpreted as a degree of laziness, or at least disinterest, on the part of those who created the scraper. There's no ideal option here, but if the user-agent string is the main identifying piece of information available, then that's what has to be used at least for the moment. – Su' Apr 4 '12 at 9:24
  • I know there's at least one more question dealing with this with a code example. I just need to find it. – John Conde Apr 4 '12 at 11:18
  • @JohnConde I see the questions have the same themes, but the duplicate question hardly supplies a canonical answer for this - surely we can and should do better, this is a big problem for some webmasters. – toomanyairmiles Apr 4 '12 at 12:05
  • They seem to be asking the same question to me. If the other question doesn't have a great answer then we need to add one to it (assuming someone can). But keeping this one open doesn't accomplish anything and puts the information in two different places which is what StackExchange doesn't want. – John Conde Apr 4 '12 at 14:12

Perisable press has a good run down on dealing with content scrapers, as does Chris Coyer at CSS Tricks the general view is do nothing and take advantage of it where you can. Summary of good advice from perishable press below...

How to Deal with Content Scrapers

So what is the best strategy for dealing with content-scraping scumbags? My personal three-tiered strategy includes the following levels of action:

  • Do nothing.
  • Always include lots of internal links
  • Stop them with a well-placed slice of htaccess

These are the tools I use when dealing with content scrapers. For bigger sites like DigWP.com, I agree with Chris that no action is really required. As long as you are actively including plenty of internal links in your posts, scraped content equals links back to your pages. For example, getting a link in a Smashing Magazine article instantly provides hundreds of linkbacks thanks to all of thieves and leeches stealing Smashing Mag’s content. Sprinkling a few internal links throughout your posts benefits you in some fantastic ways:

  • Provides links back to your site from stolen/scraped content
  • Helps your readers find new and related pages/content on your site
  • Makes it easy for search engines to crawl deeply into your site

So do nothing if you can afford not to worry about it; otherwise, get in the habit of adding lots of internal links to take advantage of the free link juice. This strategy works great unless you start getting scraped by some of the more sinister sites. In which case..

The stack network is content scraper city so it would be interesting to hear the advice of some of the high level admin's on this topic...

  • scraped backlinks are worthless – user1721135 Apr 9 '13 at 19:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.