4

Writing spiders, I have noticed that many sites will return a 403 error if I hit them from popular HTTP software libraries, unless I manually override the default User-Agent header used by the library.

For example, The Economist magazine blocks my requests if I use the default user agent headers of any Python HTTP library:

$ curl http://www.economist.com/ -A python-requests/2.9.1 --write-out "%{http_code}\n" --silent --output /dev/null
403
$ curl http://www.economist.com/ -A python-Urllib/2.7 --write-out "%{http_code}\n" --silent --output /dev/null
403

But if I fake a browser user agent, put in a nonsense user agent, or provide an empty user agent, they're happy to accept my request:

$ curl http://www.economist.com/ -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/53.0.2785.143 Chrome/53.0.2785.143 Safari/537.36" --write-out "%{http_code}\n" --silent --output /dev/null
200
$ curl http://www.economist.com/ -A '' --write-out "%{http_code}\n" --silent --output /dev/null
200
$ curl http://www.economist.com/ -A banana --write-out "%{http_code}\n" --silent --output /dev/null
200

The Economist is the biggest site I've come across with this behaviour, but certainly not the only one - this behaviour seems to be common. Why? What purpose does this blocking serve from the website's perspective? Is it a (misguided and ineffective) security measure? An attempt to get more meaningful user agents from bots? (But for what purpose?) Or does something else motivate these filters?

  • 5
    The answer is simple. There are sooooo many scrapers out there that use common libraries and what not to abuse sites. These are often easy to see. Using a recognizable agent, is one thing, but make sure you put your URL in the agent string. Some sites will block accesses that are abusive regardless of the agent name. I do this. If you tell me who you are and I agree with your mission, your agent string can be white-listed. Keep in mind, that there are far more bots than users these days and site owners are sick of it. Go gentle, be nice, and transparent about who you are and your mission. – closetnoc Oct 14 '16 at 17:16
  • 3
    @closetnoc The answer is simple ... so post it as an answer! – TRiG Oct 14 '16 at 20:59
3

This is due to the number of people who embed these HTTP libraries in their own software for the purpose of scraping content from other sites, which is often done for the purposes of copyright infringement. Well-made crawlers which are legitimate and designed for a specific purpose (like archiver bots and search bots) have their own custom user agent strings to uniquely identify them. Based on this a general feeling held by many who apply these sorts of restrictions to their own sites is that any connections using the default user agent string from these libraries have not been made for a legitimate purpose and any which are caught out by mistake would probably result in the developer contacting the webmaster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.