3

I noticed that some Nginx configurations block download agents using Wget like this:

if ($http_user_agent ~* LWP::Simple|wget) {
    return 403;
}

Why block Wget? If I use the above code in my Nginx config, will that also block downloading of PDFs and other common files that can be downloaded by genuine users from my Joomla site?

migrated from serverfault.com Oct 30 '14 at 22:37

This question came from our site for system and network administrators.

3

User agent wget

Blocking the user agent wget will block operating systems that use wget to download files within the terminal i.e Linux, MacOS, Arm etc. This will block every file from being downloaded using wget example.pdf.

But why block wget?

There's no right or wrong to this question but generally webmasters want site visitors actually visiting their pages to download their stuff, this is among one of many hot linking prevention methods.

What will get blocked?

Only the user agent wget associated with the terminal application as described previously all file types will be prevented including .html files. Browsers will not be affected unless they use the wget user agent which by default they don't and only knowledgeable users will be able to achieve this but users have no benefit to user that user agent so realistically a block will never occur in anyone's browser.

When wget is useful

It's not very useful for users to use wget to download a PDF file or video clip unless they want to mirror it or because the file size is very large and they want to store it somewhere remotely. However it can be useful for things like deployment of content management systems such as Joomla and Wordpress which have several hundred of files each and using:

  • wget http://www.example.com/cms.tar
  • tar xvf cms.tar

By doing the above actions you can deploy a content management system within seconds rather than minutes vs using SFTP or FTP method due to the limitation of the protocols. So wget shouldn't be used for things like operating systems, content management systems and so forth but generally its useless for most websites so blocking it shouldn't become a major issue.

  • 1
    Excellent answer. For sites that do not have downloads, from my experience which is long, wget accesses are used for various sorts of activities such as content scraping and so on. In fact, I just saw the other day a content scraper software that still touts using wget as it's primary feature. I like wget and there are reasons to use it. However, far too many times it is used to abuse websites including denial of service (believe it or not). As for most sites, it is not a browser like lynx that is used by people, but an offline replicator of websites which can be automated. – closetnoc Oct 31 '14 at 1:19
  • Thank you @bybe and closetnoc This is the best explanation I've seen on wget and what we gain by blocking it. It makes so much sense now – Neel Oct 31 '14 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.