Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?

share|improve this question
1  
You cannot block all the scrapers. Tons of them... –  Oleg Apr 3 at 11:59
    
Yeah. But we are getting very close. –  closetnoc Apr 4 at 0:43
    
You know that pretty much every user agent can be spoofed right? –  Braiam Apr 4 at 4:17
1  
Of course. But agent names are garbage data. The statement was you cannot block all the scrapers and I replied that we are getting closer. –  closetnoc Apr 4 at 4:47
add comment

5 Answers 5

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure

If your site is password protected properly, there shouldn't be any need to block particular user agents :) x

share|improve this answer
7  
And of course, wget lets the user set the User-Agent string (it's the -U / --user-agent option), so blocking wget's default user-agent string is almost entirely useless. –  T.J. Crowder Apr 3 at 12:57
    
It's password protected for experimenting, soon I want to make it public, and would rather have as less scrapers as possible, so if I caught that one in advance, I think it would be good to block it on .htaccess before making the site public, wouldn't it? –  getbuck Apr 3 at 13:26
1  
If the page is going to be public, you're not going to be able to stop all scrapers (as @T.J.Crowder pointed out it's easy for them to disguise their useragent anyway). If something is public, it's public - why would you want to block people accessing the content? –  Emily Shepherd Apr 3 at 13:28
    
I'm afraid that somehow a scraper would get my content indexed before me, and thus make my site look duplicate, or any other similar SEO concern –  getbuck Apr 3 at 13:32
3  
Keep in mind that Google uses bots to crawl pages too, so not all bots are bad :) You won't be able to stop people downloading your content if it's public - if you block wget people will just change its user agent, or just open up a web browser and download it themselves. This is very unlikely harm your SEO as your site will be most up to date, and Google is very clever. If you're really concerned about this, submit your site to Google Webmaster tools when you publish it - that'll give you stats about what the Googlebot thinks about your site etc x –  Emily Shepherd Apr 3 at 13:37
add comment

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string.

wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent MSIE(blahblah) and impersonate Internet Explorer if you start blocking at the HTTP level. I've written scraping scripts before and you'd better believe changing the UA is one of the first steps (if that doesn't work you could always switch gears and simply write a script to automate IE, of course).

If you're really concerned, you'll need to try and catch bot-like behavior -- pages without referrers, too many requests in too short a time, etc. However, I'm afraid you'll quickly find that it's pretty trivial for someone who wants to scrape your site to bypass any measure you could possibly take (short of those that would be too onerous to your users, like only allowing one page view per hour or something). This is also likely to be a big time sink.

Essentially, if legitimate users can see your page, there's not much you can do to keep scrapers from seeing it too.

share|improve this answer
add comment

wget is often used for scraping. It's a command-line tool to download webpages and their assets. If your website isn't being publicized, you can almost be sure that it's a bot doing scraping. So yes, you could block it, but also be aware you may need to do something more sophisticated than blocking it with robots.txt since wget can easily be told to ignore robots.txt

To this particular user agent in .htaccess, you could add the following:

BrowserMatchNoCase Wget/1.12 (linux-gnu) wget
Order Deny,Allow
Deny from env=wget
share|improve this answer
    
Do you know what's the appropriate command to add to .htaccess to block all these Wget ? –  getbuck Apr 3 at 13:24
1  
how do I hate this answer, let me count the ways: wget is not a linux tool it is a gnu tool and was written for unix and has been ported to every major operating system including OSX and Windows. wget isn't the best tool for scraping, it is barley adequate at best (it is very useful for mirroring and is also used as a dowloader for uzbl). Your certainty that it is scraping is great, can you prove it or are you just right? wget respects the robots.txt file except on single file downloads as documented in the man page. –  hildred Apr 3 at 16:17
    
@hildred How is "mirroring" a site or "downloading" a web page not "scraping"? The intention of the question was not to determine weather it's a GNU/Linux tool, it's what it's purpose is. It's not a typical web browser, and nobody is trying to navigate to his site with wget to view it as a human. Have you ever used wget to browse the web? Didn't think so. And no, I can't "prove" that it's scraping the website, but like I said nobody uses wget to browse the web, so it's the most likely conclusion. –  nathangiesbrecht Apr 3 at 17:06
    
@hildred you can also ignore robots.txt with the -e robots=off flag with wget. But again, thanks for "hating" my answer without providing any useful constructive criticism. –  nathangiesbrecht Apr 3 at 17:11
2  
... and the reason for that is that robots.txt contains instructions for spiders. Downloading a single url is not spidering. From robotstxt.org, "Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically". Downloading a single URL provided by a user is not traversing automatically, but wget --recursive is (and checks for robots.txt by default). –  Steve Jessop Apr 4 at 1:03
show 7 more comments

Every time I read questions like this I think of Kevin Spacey's character in Henry and June. The fellow who was always writing his greatest Novel, but was so worried about someone stealing his ideas he kept it locked away in a briefcase, carried close to his chest...

Every linux user is a "legit" wget user. I use it often for grabbing debs, videos, binaries, whatever. It's easy to drive from the command line so, yeah, it makes a great scraper. But that's definitely not its only use, and making it appear as firefox or msie is just one --parameter away, so you're wasting your time blocking it. If anything, if you do that you're going to attract the attention of anyone passing by; they'll change the user-agent string and start digging for what you have "hidden."

share|improve this answer
    
You half made my point for me. You are right. The agent name can be changed easily. But still, this does not mean that WGet should or should not be blocked. It is a single small step in securing against a site from abuse. Other steps have to be taken too. I advised that blocking the agent depends upon your user. I too use WGet periodically as you say you do. But I am sure you use it against sites that expect it to be used and not content only sites like blogs and such where people count on their content for revenue. It is a personal decision plain and simple. –  closetnoc Apr 4 at 0:42
add comment

Why another answer? Because both are right but with caveats. I study these things as a subject area regarding security.

WGet is an offline web browser. It is not interactive like Linx which is a text based browser that works something like any other browser. WGet is most often used to capture resources such as downloads, videos, audio files and capture and create a localized copy of a website. It can be used by scrapers, but it would awkward to do and would require hand written code to parse files. I have done this for customers. I would not recommend it.

WGet is not a scraping tool. It is generally not used by scrapers directly because there are so many software scrapers out there that it would not make sense. However, the use of WGet does allow the user to do things that would not otherwise happen. For example, I can create a resource URL corrected version of a page, part of a site, or whole site, to browse offline off of my hard drive. It is possible for WGet to download a page, part of a site, or whole site corrected for redeployment somewhere else. But from my experience, this does not happen.

WGet is not generally a tool that anyone really needs to do work against your site.

And that is exactly my point. Anyone using WGet is not a casual user and often has something nefarious in mind. But not always. I have used WGet to download research papers, data, and any other resources where I would have to click a bunch of links essentially one at a time and pay attention to the process. For example, I could specify what page to look at, and what resources I wanted to download and I could trigger it to run on my robot server completely unattended. I use WGet for valid work. But users like myself are extremely rare, and I would not be using it against your site without your knowing it.

I would block this agent. I have it blocked on all the sites I control. If you are not offering downloads videos, audio files, e-books, and other similar resources for open downloading, block this agent. The user is likely up to no good. In fact, during my study, outside of a resource site, I have not seen a valid WGet user.

share|improve this answer
2  
This is an aristocratic answer. "I am a legitimate user of wget, but you peasant are a no good nefarious user of wget and should be blocked." –  hildred Apr 3 at 16:25
4  
I think this answer gives totally the wrong impression of both wget and how data on the internet is shared. If you have made a web resource public, it is public. It is not up to you to decide "I'll let Chrome users download it, but I won't let wget users view it". You can't casually say "there are no reasons for wget hitting your site". You certainly don't need to get the webmasters permission to use wget. People should be free to use open data as they please - if you have such issues with how people use data, they shouldn't be on public facing pages. –  Emily Shepherd Apr 3 at 18:15
2  
And a big amen to @EmilyShepherd, she sees the same problem from an entirely different point of view. and shows how useless your assertion that resource has a different meaning in programming. –  hildred Apr 3 at 18:22
2  
In extension to my point above, I also believe you've missed various aspects of the web - the semantic elements of HTML5, microdata, microformats, RDF: all examples of how html pages can (and should!) be semantic enough to parse for information if people want it. Screen readers, translators, custom blogs with pingbacks are some of the most obvious uses for accessing a page not via a web browser. Maybe people might use wget for those things, maybe they wouldn't - but blanket blocking anything that "has no reason for being there" will lead to needlessly making legitimate tasks harder. –  Emily Shepherd Apr 3 at 18:26
2  
so @closetnoc, you don't want me reading your blog? my typical workflow for serious blog reading is to download the interesting and thought provoking articles, convert them to epubs and read them in the park where I have the quiet to read them and understand them. –  hildred Apr 3 at 18:26
show 23 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.