Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little.

I have implemented a robots.txt policy. I posted it below..

User-agent: wget
Disallow: /

User-agent: libwww
Disallow: /

User-agent: *
Disallow: /  

Issuing a wget from my totally independent ubuntu box shows that wget against my server just doesn't seem to work like so....

http://myserver.com/file.csv

Anyway I don't mind people just grabbing the info, I just want to implement some sort of flood control, like a wrapper or an interceptor.

Does anyone have a thought about this or could point me in the direction of a resource. I realize that it might not even be possible. Just after some ideas.

Janie

share|improve this question

migrated from stackoverflow.com Jun 29 '11 at 18:52

This question came from our site for professional and enthusiast programmers.

1  
I know how to block them through htaccess. I just don't know how to throttle them. If you want the snippet for blocking them, let me know and I'll post it. –  John Conde Jun 29 '11 at 19:23
    
That's great John! I'll take what I can get for now. It would help. –  Jane WIlkie Jun 29 '11 at 20:08
add comment

1 Answer

If you decide you want to block wget and libwww you can either redirect them to a page telling them why you're blocking them with this code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww
RewriteRule ^(.*)$ http://www.example.com/blocked.html

Or you can flat out reject their request with this code:

SetEnvIfNoCase user-agent  "^wget " bad_bot=1
SetEnvIfNoCase user-agent  "^libwww" bad_bot=1
<FilesMatch "(.*)">
  Order Allow,Deny
  Allow from all
  Deny from env=bad_bot
</FilesMatch>

Just place either snippet in a .htaccess file in your root directory or the directory where the files they are downloading from are.

I have used the second snippet to block bots from a site I had that was getting scraped a lot. I haven't used the first snippet but it looks like it should work well if you choose to go that route.

share|improve this answer
    
Or if you are running IIS 7+ you can use URL Rewrite as well, just place the following XML in a web.config file. <configuration> <rewrite> <rules> <rule name="Block Robots"> <match url="^(.*)$" ignoreCase="false" /> <conditions logicalGrouping="MatchAny"> <add input="{HTTP_USER_AGENT}" pattern="^wget" ignoreCase="false" /> <add input="{HTTP_USER_AGENT}" pattern="^libwww" ignoreCase="false" /> </conditions> <action type="Redirect" url="http://www.example.com/blocked.html" redirectType="Found" /> </rule></rules> </rewrite>. –  CarlosAg Jun 30 '11 at 1:11
1  
@CarlosAg Make that an answer so we can upvote it –  John Conde Jun 30 '11 at 2:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.