I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little.

I have implemented a robots.txt policy. I posted it below..

User-agent: wget
Disallow: /

User-agent: libwww
Disallow: /

User-agent: *
Disallow: /  

Issuing a wget from my totally independent ubuntu box shows that wget against my server just doesn't seem to work like so....


Anyway I don't mind people just grabbing the info, I just want to implement some sort of flood control, like a wrapper or an interceptor.

Does anyone have a thought about this or could point me in the direction of a resource. I realize that it might not even be possible. Just after some ideas.


  • 1
    I know how to block them through htaccess. I just don't know how to throttle them. If you want the snippet for blocking them, let me know and I'll post it.
    – John Conde
    Jun 29, 2011 at 19:23
  • That's great John! I'll take what I can get for now. It would help. Jun 29, 2011 at 20:08

1 Answer 1


If you decide you want to block wget and libwww you can either redirect them to a page telling them why you're blocking them with this code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww
RewriteRule ^(.*)$ http://www.example.com/blocked.html

Or you can flat out reject their request with this code:

SetEnvIfNoCase user-agent  "^wget " bad_bot=1
SetEnvIfNoCase user-agent  "^libwww" bad_bot=1
<FilesMatch "(.*)">
  Order Allow,Deny
  Allow from all
  Deny from env=bad_bot

Just place either snippet in a .htaccess file in your root directory or the directory where the files they are downloading from are.

I have used the second snippet to block bots from a site I had that was getting scraped a lot. I haven't used the first snippet but it looks like it should work well if you choose to go that route.

  • Or if you are running IIS 7+ you can use URL Rewrite as well, just place the following XML in a web.config file. <configuration> <rewrite> <rules> <rule name="Block Robots"> <match url="^(.*)$" ignoreCase="false" /> <conditions logicalGrouping="MatchAny"> <add input="{HTTP_USER_AGENT}" pattern="^wget" ignoreCase="false" /> <add input="{HTTP_USER_AGENT}" pattern="^libwww" ignoreCase="false" /> </conditions> <action type="Redirect" url="http://www.example.com/blocked.html" redirectType="Found" /> </rule></rules> </rewrite>. Jun 30, 2011 at 1:11
  • 1
    @CarlosAg Make that an answer so we can upvote it
    – John Conde
    Jun 30, 2011 at 2:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.