I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
Add a comment
|
2 Answers
Yes and No. Let me explain.
Any user can change the User Agent string that Wget uses. If the string is not changed, then Wget can easily be captured using the following in your .htaccess file.
RewriteCond %{HTTP_USER_AGENT} wget.* [NC]
RewriteRule .* - [F,L]
However, if the User Agent string is changed, then you may never know that it is Wget.
That being said, one thing is clear to any web master who has been around for a while - the log file must be examined often for abusive activity. It is possible to block bad actors as they come along and yet impossible to block them ahead of time though you can get a lot of them.
You will be able to stop most all abuses if you watch your site access logs often enough and know how to use .htaccess and regular expressions. It is not a difficult process and should be well understood by any web master.
While some will argue purely from a philosophical point of view, the fact is that Wget should be blocked in most cases. In all the years I have dealt with the web (and that is a lot more than almost anyone), Wget has no purpose to a user except to scrape resources from a website. While some sites do open themselves up to this form of activity and actually invite it, all accesses I have experienced with Wget have been a form of abuse or theft.
-
Thank you. So as I understand it, it's possible to an extent as long as I watch the logs like a hawk but impossible to prevent it before it even happens? Dec 23, 2015 at 15:31
-
@ECHELON I was talking about bad bots in general at that point. It is impossible to say when and what bad bot will visit your site, but you can get a feel for what to expect for your site by watching your log files. Like a hawk might be overkill ;-) but daily or near daily to see what kind of activity you get makes sense. Traffic changes all the time, however, you will eventually learn how to block most abuse against your site. You can also simply use something like ModeSecurity which will help a lot!! Dec 23, 2015 at 16:16
Most bots don't use wget but rather a crawler, and you can advise them to go away by publishing in your site a robots.txt file. There are some rogue bots that won't honor your robots.txt and these will have to be explicitly blocked. You can identify those by reviewing your web server's logs, but also by applying publicly available blacklists.
