Sign up ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little.

I have implemented a robots.txt policy. I posted it below..

User-agent: wget
Disallow: /

User-agent: libwww
Disallow: /

User-agent: *
Disallow: /  

Issuing a wget from my totally independent ubuntu box shows that wget against my server just doesn't seem to work like so....

Anyway I don't mind people just grabbing the info, I just want to implement some sort of flood control, like a wrapper or an interceptor.

Does anyone have a thought about this or could point me in the direction of a resource. I realize that it might not even be possible. Just after some ideas.


share|improve this question

migrated from Jun 29 '11 at 18:52

This question came from our site for professional and enthusiast programmers.

I know how to block them through htaccess. I just don't know how to throttle them. If you want the snippet for blocking them, let me know and I'll post it. – John Conde Jun 29 '11 at 19:23
That's great John! I'll take what I can get for now. It would help. – Jane WIlkie Jun 29 '11 at 20:08

1 Answer 1

If you decide you want to block wget and libwww you can either redirect them to a page telling them why you're blocking them with this code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww
RewriteRule ^(.*)$

Or you can flat out reject their request with this code:

SetEnvIfNoCase user-agent  "^wget " bad_bot=1
SetEnvIfNoCase user-agent  "^libwww" bad_bot=1
<FilesMatch "(.*)">
  Order Allow,Deny
  Allow from all
  Deny from env=bad_bot

Just place either snippet in a .htaccess file in your root directory or the directory where the files they are downloading from are.

I have used the second snippet to block bots from a site I had that was getting scraped a lot. I haven't used the first snippet but it looks like it should work well if you choose to go that route.

share|improve this answer
Or if you are running IIS 7+ you can use URL Rewrite as well, just place the following XML in a web.config file. <configuration> <rewrite> <rules> <rule name="Block Robots"> <match url="^(.*)$" ignoreCase="false" /> <conditions logicalGrouping="MatchAny"> <add input="{HTTP_USER_AGENT}" pattern="^wget" ignoreCase="false" /> <add input="{HTTP_USER_AGENT}" pattern="^libwww" ignoreCase="false" /> </conditions> <action type="Redirect" url="" redirectType="Found" /> </rule></rules> </rewrite>. – Carlos Aguilar Mares Jun 30 '11 at 1:11
@CarlosAg Make that an answer so we can upvote it – John Conde Jun 30 '11 at 2:34

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.