37

I found a website that provides a huge filterable table with up-to-date data about cities in a country. This site uses an infinite scrolling approach to load the rows for the table.

By exploring the site using Chrome's Developer Tools, I found it makes AJAX requests to some internal URL to get the data. This URL includes a lot of query parameters corresponding with the filters.

I tried to access that URL directly in my browser and I am getting all the data in a nice JSON format. I can even play around with the filters to get the concrete data I need. This URL is actually accessible, so I do not need to do anything hacky, I'm just calling a URL that is there in the net.

So my question is: how legal or illegal is it for me to use that URL to retrieve the data for my own purposes?

Note: I do not want to create the same kind of cities list, but I want to use that data to create a little online game, potentially to earn a little money...


IMPORTANT Notes about some responses and comments

This is just an example scenario, I'm not looking for a place to grab data about cities. Consider a website with data about updated football players' performance in a season if you want.

As for the concrete country, again, I'm not thinking in a particular legislation, if you know a country where is clearly (il)legal, that'll be useful info.

  • 4
    What are you planning to do with this data? It doesn't sound like getting the data would be illegal, but copying it or displaying it to users might violate copyright – Stephen Ostermiller Dec 11 '14 at 16:09
  • 2
    The site from which you are using the API might use technical rather than legal means to prevent you from using this API. It is usually better to get permission to use an API to ensure that it doesn't get blocked and inconvenience your users. – Stephen Ostermiller Dec 11 '14 at 16:11
  • 3
    It's simple! ask a licensing consultancy or a copyright lawyer! not a webmaster! – Simon Hayter Dec 11 '14 at 16:43
  • 6
    @bybe, I'm sorry I cannot find the Stack Exchange's Licensing Consultancy site, and I only ask questions to Stack Exchange users or to god... and god never responds. – MikO Dec 11 '14 at 16:49
  • 1
    You will want to read my answer and think again carefully. – closetnoc Dec 11 '14 at 19:00
27

Let me be clear. There is one thing I know fairly well, it is copyright law. I am not a lawyer, however, knowledge of copyright was a constant requirement of my consultancy for 30 years. As an added bonus, I consulted primarily to telecos and often worked with subscriber data and data analysis and presentation of said data for sale and re-use. I am at least, uniquely qualified to answer this question on this forum.

I will explain this the best that I can by: one, defining proprietary verses ordinary means; two, defining the cited case exception and other related copyright considerations; and three, being clear on the answer.

Let me clarify copyright some. The example of a phone book is a misnomer. When you get a telephone, you have entered into a private contract agreement as a private citizen with a private company and the resulting information, made public or not, is private proprietary data and therefore the contents of a phone book is proprietary (pay attention to this word) simply because it cannot be obtained generally through any other means except through company data sources- the subscriber data. If data can be derived through ordinary means, such as walking around and writing down house numbers and street names, then that is publicly available data and clear to use. This is not to say that telephone numbers cannot be obtained through ordinary means. It can be.

To clarify further. To quote from: http://www.lib.umich.edu/copyright/facts-and-data

In no case does copyright protection for an original work of authorship extend to any idea, procedure, process, system, method of operation, concept, principle, or discovery, regardless of the form in which it is described, explained, illustrated, or embodied in such work.

This paragraph is misleading. This exception described in this paragraph is covered by patent and other laws. Copyright only extends to the creation of a work.

The:

“sweat of the brow” doctrine

...refers to any activity such as going house to house and gathering the data manually. This is the definition of ordinary means. It is possible to knock on doors and ask for the same telephone data. Only in as much as you can gather the facts by ordinary means is that data or portion of the proprietary data public.

The ordinary way around using telephone data is to: one, obtain the original data through legal means; and two, apply the fair use doctrine. This would entail getting a copy of the phone book directly from the company which may be free or for a charge, and organizing the facts within in a different way as to create a new work. Have you tried to get a Seattle phone book when you are in Chicago? You will find that the telephone company will likely charge you a surprising fee for it. However, if you are a telephone subscriber in Seattle and you ask for a Seattle phone book, the fee would be far less or even free. I have had to do this many times. There are people who's job it is just to obtain telephone books from telecos in person and paying the fee if required.

The ruling cited in case Feist Publications v. Rural Telephone in the above link (in this answer) hinges on two facts: one, being that the data by the rural cooperative operator as a local monopoly was required to be made publicly available by operational agreement; and two, that the presentation of the work was copyrighted and not the facts contained within due to fact #1. Therefore, only within narrow parameters can this case be considered as a precedent case and must be discarded. Ordinarily, private company subscriber data is not required by agreement to be made public. You have to remember that rural cooperatives are established as public trusts/entities for the public good and owned by the public and/or cooperative members and therefore operate under legal restrictions that allow it to be approved to operate or exist. Each case is different. The citing of the above case (on the linked page) as an argument without explanation of the carve-out exceptions is misleading.

In the early days of the Bell Telephone company, the company was required as a monopoly to make telephone data public unless restricted by the subscriber. When the Bell company was split into the baby Bells, Bell Atlantic, Bell South, and so on, these companies were still required as monopolies to make telephone data public as defined before. But with deregulation and indeed with VoIP, cellular, and other options, monopolies are rare. Only in monopoly scenarios can the above cited argument be made.

Continuing to cite the link above (in this answer):

Just because data is not protected by copyright, does not mean there are not other legal considerations that may come into play when you wish to use someone else’s dataset.

Keep this in mind.

Any given dataset and the presentation there of, regardless of the data origin, is a work unto itself. The public presentation of the facts, irregardless of the means, is a work unto itself.

Given that you are not obtaining the data through ordinary means, even though the data is made public, and regardless of the original origin of the data, it is not free to use as you described and you could be criminally charged and held civilly liable for potential copyright infringement as well for criminal trespass and illicit use of computer and other communications equipment not ordinarily authorized and can fall under RICO statutes.

Is it legal to use? No! Absolutely not! It was not obtained through ordinary means nor is it likely the intent of the website operator to expose proprietary data. Any absence of an AUP (acceptable use policy) will not help you. There are assumptions made under the law as to the "reasonable man", "reasonable standard", and "reasonable assumption" that protects the website owner in this case. It is not reasonable that a clever person would use a "vulnerability in the design/creation" of the website to obtain data for other use. As well, if the site profits from it's activities, further protections come into play.

  • 3
    Great counterpoint. It bears repeating that it is often worth consulting a lawyer when in doubt about the legality of what you are doing. – Stephen Ostermiller Dec 11 '14 at 19:19
  • 1
    The NFL states that it exclusively owns the team/player stats and therefore licenses the data for use. If I see it on T.V., it through a license agreement, if I see it in the news paper, it is through a license agreement, magazines, same thing. Most sources you would get the data would be by license. However, if obtained through ordinary means, such as asking a friend, then that is legal. However, the NFL, while very careful about licensing, will not shoot you if you post stats on a website that does not profit from the information. They may write a letter, but generally, they will not bother. – closetnoc Dec 11 '14 at 19:34
  • 2
    The law in the UK is not the same in that a collection of "public data" can be copyrighted, even if each data item can not. – Ian Ringrose Dec 12 '14 at 12:30
  • 2
    @closetnoc, is the illegality just in using a hidden API to obtain the information? If you were to use the website as intended and to write down the information manually, and then use it, would that be legal? Followup: if so, and you automated that process, would it still be legal? – Jason Dec 12 '14 at 19:05
  • 1
    @closetnoc I'm curious: if, hypothetically, the site's robots.txt did not restrict the URL, and queries from it were available on various search engines, would that affect your analysis? With the JSON I've seen being more readable than the touted "human-readable" XML (but they said! :D) [e.g. SOAP] do you agree that it would be arguably "obtained by 'ordinary means'" at that point? On a browser that had one of the easy-JSON-viewing extensions installed, clicking a link and viewing the data would be trivial. And I imagine an auto-JSON-REST-query-field-search addition not to be far behind... – shelleybutterfly Dec 12 '14 at 20:20
6

One thing that doesn't seem very clear in the other answers here...

Whether it's "legal" or not, first and foremost, depends on the country. If we're talking about the United States, for example, then using the data itself is not illegal. However, I'd advise you to use the real data from the US Census. They offer tons of data through what they call TIGER products. This data set is the same data set that GIS professionals use to populate Bing maps, Google maps, etc.

However, while the data may be freely available, that does not necessarily mean the data from this exposed API is legally available. You say it's in JSON form, which suggests it's been 'massaged' from its original format into this format - and that custom format could fall under intellectual property. That, I believe, would be illegal to use unless you have the license to use it. Like others here, I am not a lawyer, but the company doesn't even need to point the finger at you and call you a hacker. Proprietary data is proprietary data, even if it is handed out unintentionally. You should contact the company and let them know all of this data is exposed to the outside world, and ask for permission to use it. Without doing that, and by having this question on stack exchange as evidence, it'd be easy to build a case against you. You've essentially said "This doesn't look legit, but I like it anyway and I want to make money off of it." Again, I'm not a lawyer, but that doesn't look like a great way to start a trial.

The thing is, though, if you're interested in city names and other geographic data, almost all of it is freely available, regardless of country. Last I knew, the US publishes the most data, but there's data out there for virtually every country. I'm hesitant to say all only because I'm a programmer and proving a "for all" statement is hard...if you pick an arbitrary country, the chances are better than good that the data is out there. If you have a specific country in mind, head to the GIS Stack Exchange. The main thing you're looking for are called "shapefiles", so ask a question like "Where can I get shapefiles for __________?" There's also OpenStreetMap which is an open source map. I'm not sure how easy it is to get their shapefile data, but if you can get it from them(and I don't see why you wouldn't be able to, you're able to run offline maps based on locally stored information), then you have all the data you need and you're in the clear legally. You'll have to spend time massaging the massive amounts of data down to what you want, but shapefiles are always very-well defined and easy to parse.

  • 1
    You bring up some great points! Any data from the U.S. Government by legal authority is public domain as it has already been paid for and owned by the public through taxes. The government likes to charge fees for this data when provided in particular format such as printed (GPO government printing office), on magnetic tape or CD or other media, and so on. They do this to recover labor/material costs, though I sometimes argue over the fee vs. cost. The U.S. Government has been in the profit game for a couple of decades. Why do you think they really want all the extra census data? They sell it. – closetnoc Dec 11 '14 at 22:15
5

While closetnoc has discussed the issue of the data itself, there's a larger legal concern: you are not authorized to access the API offering the data.

The baseline for most computer crime laws involves the notion of "unauthorized access to a computer system". You should not confuse this reference to authorization in the legal sense with the concept of authorization when it comes to access control. The owner of a system does not have to secure his system for access to it to be illegal, just as you are still trespassing when you enter a house with an unlocked door.

In this case the apparent lack of security does not imply an authorization to use it. The concept of the internet has little precedent so far in case law but you can imagine the use of the HTTP 80 protocol to imply public authorization to view a website. Contrarily, background RPC protocols (even if they might run on HTTP requests) are not typically understood to be publicly available unless the operator publishes the service as such, granting authorization for use to third parties.

So ongoing use of the API to retrieve data would be illegal. The act of taking a data dump from the API to build your own dataset would also be illegal. Whether use of the data after that is illegal is a giant grey area but closetnoc has covered most of the concerns.

Of course if you modify the data dump after the fact to be unrecognisable it will be next to impossible to prove that you committed a crime. But if you're going to that much trouble why not source the data from a legal source instead?

  • Interesting points and I agree with you, except it doesn't make sense to me when you argue "the AJAX RFC protocol is an internal system not intended for public access". I'm not sure what protocol has to do with it. Many organizations offer exposed data services such as this for the public's use. For instance, local governments offering GIS data. It makes more sense to me that significant facts are the way OP discovered the data service and that it is apparent it isn't intended for public use. Am I off base here? – pseudocoder Dec 12 '14 at 15:28
  • I do cover the fact that access of this type would not be legal under any standard, however, I am glad you mentioned it again. It is an important point. Any AUP should state that access authorization by default is NOT granted (of course it does not have to be that draconian). The idea is to establish a negative default first then define the acceptable use in a rather narrow webbie kinda way. – closetnoc Dec 12 '14 at 15:46
  • 1
    @pseudocoder No you are correct. What I meant by that is that there is an established reasonable expectation that access to a HTTP service is public by default while the opposite is true for RFC services and similar protocols. Such services are generally published for public use and announced as such. As I said I don't know the case law on this or how the various cybercrime laws handle this in practice, but it's an important distinction. [...] – Lilienthal Dec 12 '14 at 16:15
  • 3
    If you didn't have this expectation of public access you'd be in violation of accessing StackExchange because its owner hasn't explicitly granted you the authorization to view the page. A reasonable person (a popular concept in law discussions) would expect a website service to be accessible to all and for sensitive parts to be hidden behind an access control layer. That same person would NOT expect "hidden" background services (that the average person doesn't understand) to similarly qualify as free for all. I've edited my answer to hopefully better explain this dichotomy. – Lilienthal Dec 12 '14 at 16:18
  • By "RFC" do you mean "RPC"? – ruakh Dec 13 '14 at 4:21
4

It probably depends on the nature of the data. Pure data (think telephone directory) cannot be copyrighted. So a list of cities from an API should be fair game to copy and show to users. However, if that API has descriptions of the city those descriptions would fall under copyright law and you wouldn't be able to use them without violating copyright.

If you can legally copy the data, I would recommend copying it to your own site to prevent your API usage from being shut down prematurely.

  • 1
    Thanks for your reply. Just not clear what can be pure data. Some examples: the number of inhabitants in a city or the number of new vehicle registrations last year in a city... Or in other contexts: the number of goals scored by Cristiano Ronaldo this season or the list of concerts of U2 next year... are all those pure data? – MikO Dec 11 '14 at 16:27
  • I am not a lawyer. An intellectual property lawyer would be better suited to advise you as to how the law might be applied to your specific data. The examples you gave look like data to me. Only the presentation or arrangement of them can be copyrighted (per the link in my answer) – Stephen Ostermiller Dec 11 '14 at 16:38
  • 1
    I believe that case law supports the fact that the names and telephone numbers in a phone book are not subject to copyright protection. See Feist v. Rural: "The white pages of a telephone book did not satisfy the minimum originality required by the Constitution to be eligible for copyright protection, and effort and expenditure of resources are not protected by copyright." – Stephen Ostermiller Dec 11 '14 at 17:32
3

(IANAL, and laws and norms vary widely throughout the world, but certain things tend to remain consistent due to IP treaties. If you have a professional issue outside of your specialty, consult with a profesional.)

Generally, legally, an API is not considered to be "intended for public consumption" unless it's actively documented as a public API, with specified terms of service. The fact that the public can reach the API does not make it public.

In cases where the status of the data itself isn't starkly public-domain, and in a few cases where it clearly is public domain, the intent of the entity providing the API matters a great deal. If the website operator intended the API to be used to feed a dynamic webpage, or a mobile application (to name two common examples) any other usage is "unauthorized" unless specifically authorized somewhere. If the intended consumer was a snippet of dynamic code in a specific webpage, for the purpose of placing human-understandable pixels on a display in a specific, (hopefully)pleasing and useful manner, any other usage is unauthorized.

The technical ability to enter a building through an open window without opening or breaking anything won't protect you from being arrested for criminal trespass...

Also, it is almost never wise to play "technical ability" vs "original intent" games with an intellectual property lawyer. If nothing else, remember that the lawyers who consistently lose those cases don't keep getting paid for them.

2

What you are talking about is likely fine. You are hyperlinking to information then transforming it.

In Perfect 10, Inc. v. Amazon.com, Inc.,[19] the Ninth Circuit again considered whether an image search engine's use of thumbnail was a fair use. Although the facts were somewhat closer than in the Arriba Soft case, the court nonetheless found the accused infringer's use fair because it was "highly transformative." The court explained:

We conclude that the significantly transformative nature of Google's search engine, particularly in light of its public benefit, outweighs Google's superseding and commercial uses of the thumbnails in this case. … We are also mindful of the Supreme Court's direction that "the more transformative the new work, the less will be the significance of other factors, like commercialism, that may weigh against a finding of fair use."

In addition, the court specifically addressed the copyright status of linking, in the first US appellate decision to do so:

Google does not…display a copy of full-size infringing photographic images for purposes of the Copyright Act when Google frames in-line linked images that appear on a user's computer screen. Because Google's computers do not store the photographic images, Google does not have a copy of the images for purposes of the Copyright Act. In other words, Google does not have any "material objects…in which a work is fixed…and from which the work can be perceived, reproduced, or otherwise communicated" and thus cannot communicate a copy. Instead of communicating a copy of the image, Google provides HTML instructions that direct a user's browser to a website publisher's computer that stores the full-size photographic image. Providing these HTML instructions is not equivalent to showing a copy. First, the HTML instructions are lines of text, not a photographic image. Second, HTML instructions do not themselves cause infringing images to appear on the user's computer screen. The HTML merely gives the address of the image to the user's browser. The browser then interacts with the computer that stores the infringing image. It is this interaction that causes an infringing image to appear on the user's computer screen. Google may facilitate the user's access to infringing images. However, such assistance raised only contributory liability issues and does not constitute direct infringement of the copyright owner's display rights. …While in-line linking and framing may cause some computer users to believe they are viewing a single Google webpage, the Copyright Act, unlike the Trademark Act, does not protect a copyright holder against acts that cause consumer confusion.

State of US law after Arriba Soft and Perfect 10

The Arriba Soft case stood for the proposition that deep linking and actual reproduction in reduced-size copies (or preparation of reduced-size derivative works) were both excusable as fair use because the defendant's use of the work did not actually or potentially divert trade in the marketplace from the first work; and also it provided the public with a previously unavailable, very useful function of the kind that copyright law exists to promote (finding desired information on the Web). The Perfect 10 case involved similar considerations, but more of a balancing of interests was involved. The conduct was excused because the value to the public of the otherwise unavailable, useful function outweighed the impact on Perfect 10 of Google's possibly superseding use.

Moreover, in Perfect 10, the court laid down a far-reaching precedent in favor of linking and framing, which the court gave a complete pass under copyright. It concluded that "in-line linking and framing may cause some computer users to believe they are viewing a single Google webpage, [but] the Copyright Act . . . does not protect a copyright holder against acts that cause consumer confusion."

Emphasis mine. linky

You are simply using a hyperlink, you aren't making a copy, you aren't displaying a copy, you aren't harming business interests, and you are highly transforming it. I'd say youhave every element needed to be fine. But, IANAL.

  • The OP's question is not about a hyperlink. Nor does the case you cited apply. In fact, this case should not have been filed. The copyright infringement accusation made by the claimant is nonsensical since the fair use doctrine would apply squarely. The keyword is transformative. This is another key element that defines fair use along with the right to reference or quote another's work which search engines do. The OP is talking about whole sale extraction and use of another's work which is a violation of copyright amongst other RICO violations which are federal and significant. – closetnoc Dec 12 '14 at 20:41
  • /* I AM NOT A LAWYER / i definitely ~tend~ to this side on this one... *especially via linking via http/https, of a RESTish JSON Query API, already being used to grab data for a website that's open to the public in the U.S. by current law [enough qualifying? xD] "should" not against the law due to: 1. no DMCA circumvention - the fact that no "circumvention" is necessary as there are no measures in place to circumvent. the mere claim that one has put protections on something in order to prevent people copying it, is not enough, the measures law.cornell.edu/uscode/text/17/1201 – shelleybutterfly Dec 12 '14 at 21:04
  • /* I!=AL / and 2. one of the main focii here: *linking is not the same as having. it's likely ok for an online game, as mentioned in the Q, if the player's browser downloaded; OR (maybe) if everything is grabbed in response to player acts. [IMO, it would be right to credit the site, somewhere.] but, here's the rub: we get the tech. But many many judges/juries are tech illiterate, and some seem to willfully misunderstand. At best when you're in that situation you have to waste resources fighting. So, if you can do a game like this, save yourself the trouble. Do something else. IMO. gl. – shelleybutterfly Dec 12 '14 at 21:05
  • @closetnoc: You may want to reread the OP. He is talking about visiting a web page that contains some text. That's a hyperlink. Wikipedia on hyperlinks: "An inline link displays remote content without the need for embedding the content. The remote content may be accessed with or without the user selecting the link. An inline link may display a modified version of the content; for instance, instead of an image, a thumbnail, low resolution preview, cropped section, or magnified section may be shown." That is exactly what the OP is talking about, no? – Shane Dec 12 '14 at 21:26
  • @closetnoc: He plans on taking that text used for presenting information and transforming it into a game. That's transformative. Unless he saves the data himself, he also isn't talking about extracting anything, just viewing it and transforming it. I have no idea why you think that laws against organized racketeering have any relevance here. The OP is asking if it is violates copyright laws to visit a hyperlink, or to transform the data provided by that hyperlink into something new. – Shane Dec 12 '14 at 21:27
-1

It is legal as long as you do not have to enter a password to get it, but some less sophisticated companies may claim hacking and sic a lawyer on you anyways. You must be prepared to defend yourself. You will be found not guilty, as they are publishing their data to the public, but it might cost to defend yourself. Prosecutors and cops defer to corporations. This happens often in security notices, where someone will notify a company of a security hole then the company will charge them with hacking. The company is also free to change the API without notice, possibly breaking your app.

http://www.extremetech.com/computing/146323-canadian-college-expels-student-for-white-hat-security-probing

Stop me if you’ve heard this before: A technology enthusiast gets slightly overzealous in checking for security holes, finds a significant vulnerability, comes forward with the information, and legal and personal threats are then made.

http://www.bostonglobe.com/metro/2014/03/29/the-inside-story-mit-and-aaron-swartz/YvJZ5P6VHaPJusReuaN7SI/story.html

The intruder was lurking somewhere on the MIT campus, downloading academic journal articles by the hundreds of thousands.

  • 1
    Indeed. Auernheimer spent over three years in prison for screen-scraping AT&T, even though the conviction was eventually overturned. – Andrew Medico Dec 11 '14 at 21:26
  • 4
    -1. This is incorrect and dangerous advice. The baseline for most computer crime laws involves the notion of "unauthorized access to a computer system". Just because a system is (apparently) not secured does not mean that you are authorized to use it. You should also not confuse authorization in the legal sense with authorization in an access control context. – Lilienthal Dec 12 '14 at 10:43
  • 1
    @Lilienthal The availablitity of the web site in the OP question suggests that everybody is authorized. Actually, one could even say that the user agent runs a javascript from the server and so in reverse authorizes the web site to run code and execute additional queries on the user's computer – Hagen von Eitzen Dec 12 '14 at 23:18
  • 1
    @HagenvonEitzen It most certainly does not. The website is public-facing and consumes the back-end service to display data. Compare it to a coffee shop: you're allowed to order an espresso but you're not allowed to jump behind the counter and brew it just the way you want it, you have to go through the barista. – Lilienthal Dec 12 '14 at 23:49
  • 1
    @Chloe By all rights, this answer should be down-voted simply because it is factually incorrect and may potentially help lead someone to commit an illegal act. Would you consider editing your answer? – closetnoc Dec 13 '14 at 5:31

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.