Sign up ×

I've contacted the author of this NYC homicide map from the NY Times several times to see if he could send me a spreadsheet of information the data is based off of, but I have not had any response (for a very long time).

Is there anyway I could scrap the data from the google map directly? I downloaded the swf file and converted to xml, but even within there I can not tell where the information is I want. Here is the zipped up file of the converted xml.

I'm hopeing someone with more knowledge of online mapping can quickly tell me if this possible or not (or where I should be looking in the xml file).

share|improve this question
Wow, interesting to see how it all works. What about the underlying data from ? I suspect this has something but stumped as to what to do with it?… –  Willy Mar 7 '12 at 11:18

4 Answers 4

up vote 12 down vote accepted

Quickly glancing at the page in Firebug and looking at the network calls, you can see where they are pulling the data from. Seems to be a couple of XML files, namely:


share|improve this answer
Very cool, thank you. If you feel so inclined, I would appreciate alittle more hand holding about how you figured such information out. –  Andy W Mar 5 '12 at 13:35
Basically, if you have Firefox, get the Firebug add-on. Visit the page and open Firebug. Refresh and look at the "Network" tab and you will see all of the calls the page is making, including the links above. –  ericoneal Mar 5 '12 at 14:31
I just would like to remind you that this file appears to have people's names in it and therefore their privacy is being violated? –  George Mar 6 '12 at 12:21
@George, the data is publicly available within the google map already listed on the Times Web-page (tool-tips provide all of the information). It just isn't in a format that allows me to readily manipulate the data otherwise. If your concerned about privacy your beef is with the NY Times. –  Andy W Mar 6 '12 at 13:27
@AndyW you are correct, it is with them. It seems wrong to have their names publicized. They should remove that from their website (at least the names, in the json format). –  George Mar 8 '12 at 17:16

+1 to @ericoneal's answer, but for the sake of noting an alternative approach, you could also download and install Fiddler. Fiddler routes your port-80 traffic through a proxy and provides you an interface for poking-around in the HTTP responses that follow your web request.

I'll describe the usage. In the screenshot, I just launched Fiddler, then opened your link in IE. All the data starts streaming-in without my doing anything else. Once it's settled, at left, I clicked on one of the returns (map_feed_incidents.txt, as noted by Eric), then at top-right, I select Inspectors. The pane at bottom-right provides several inspection formats. I tried a few, and the screen shows the TextView.

At a glance, the content appears to be line-break and tab-delimited (it's definitely not real XML). The top line specifies the file format, and every other line is an incident record. Here's the top-line and first record from the _incidents file (scroll right and note the id field):

LAT:DOUBLE  LONG:DOUBLE incident_date:STRING    incident_time:STRING    boro:STRING num_victims:INTEGER primary_motive:STRING   id:INTEGER  weapon:STRING   light_dark:STRING   year:INTEGER
40.665626   -73.909699  01/01/08    02:09   Brooklyn    1       7   gun D   2008

The lat/long is obvious. The other two files (_victims and _perpetrators) use the same approach. Here's the top line and first record from the _perps table:

incident_id:INTEGER sex:STRING  race:STRING age:INTEGER
7   M   B   20

The presence of incident_id is useful. Both _victims and _perps have this column, and it relates their data back to the geo-tagged _incidents table using that table's id column.

enter image description here

As an aside.. I have to agree with George and wonder why they included the victim's name. That seems like a major ethical oversight. While it's meaningless as a mapped attribute, I would not be surprised to see the perpetrator's name. But the victim's? At first I thought this may have been an unused element in the data payload, but it's really in the map?!?! That's a very questionable decision, and it leads me to believe nobody is using that map. Otherwise I think some criticism would've emerged from the general public.

share|improve this answer

I don't know if you can get the exact same data from the NYC Open data repository, but here is a link.

A slightly different approach could be to try to gather the data using the New York Times API:

share|improve this answer
It is not available from the NYC Open data repository. I believe it is culled from NY Times articles themselves on reported homicides. The NYPD does not release any dataset about homicides to that level of specificity. –  Andy W Mar 4 '12 at 20:59
They do have a "geographic" API in that list, but no documentation for it (I assume not up and running?) May be worth leaving the answer up just for that info about the Times API, it would be very cool if eventually they have a full semantic GIS api (although that is just pure speculation). –  Andy W Mar 4 '12 at 21:19

How about not doing this? Are you sure that using their data for your purposes is allowed by New York Times? Or are you convinced that the data is not copyrightable?

I quickly glanced at the source and the bottom of the page lists it as Copyright to the New York Times.

If you did this to my site then I would be extremely angry.

Update 2012-03-10 The questioner claims that this is legal for two reasons: that the information is data and not subject to copyright; and that they wish to use this for research purposes and thus copyright does not apply to them.

Copyrightability of data

US case law is a little different to many other countries in how far it goes in defining data as not being copyrightable. The leading case in the area is Feist v. Rural. The subject matter was Rural refusing to license its telephone listings to Feist so Feist used them anyway. Rural sued for copyright infringement and lost. Quoting from the case:


Canadian law is not relevant to the current question but I'll raise it to compare and contrast. The leading case here is CCH Canadian Ltd vs Law Society of Upper Canada. Quoting Wikipedia's quoting of the judgment:

McLachlin rejects Justice O'Connor's "minimal degree of creativity" test but agrees with her assessment of the "sweat of the brow" approach and finds it too low a requirement. Instead, McLachlin takes the middle ground by requiring "that an original work be the product of an exercise of skill and judgment" where "skill" is "the use of one's knowledge, developed aptitude or practised ability in producing the work" and "judgment" is "the use of one's capacity for discernment or ability to form an opinion or evaluation by comparing different possible options in producing the work". (para 16) As well, "[t]he exercise of skill and judgment required to produce the work must not be so trivial that it could be characterized as a purely mechanical exercise." (para 16) Importantly, it is required that the work "must be more than a mere copy of another work." (para. 16) However, "creativity is not required to make a work 'original'." (para. 25)

Use of copyrighted works for research purposes

In copyright law, fair use (US) or fair dealing (common law countries) recognizes that a work shouldn't exist all by itself but becomes part of the culture by being discussed, criticized, referenced, etc.

In US law, "research" is specifically allowed as a reason to for fair use but note that one of the four factors is "the amount and substantiality of the portion used in relation to the copyrighted work as a whole".

One consideration is whether the new work adds something different than the old. Quoting Wikipedia again, "To justify the use as fair, one must demonstrate how it either advances knowledge or the progress of the arts through the addition of something new. A key consideration is the extent to which the use is interpreted as transformative, as opposed to merely derivative."

Indeed, my preparation of this extended answer with lots of quotes comes under fair use (in my opinion). I'm welding a bunch of information together to create a new work.

Legality of site scraping

There was a recent case in the BC Supreme Court, Century 21 v Rogers, which discussed similar issues to what you propose. Quoting Michael Geist's comments,

Century 21's terms prohibited copying or scraping its content. By doing so, Zoocasa breached the contract...copying the full length property descriptions for several months constituted copyright infringement copying truncated, shorter descriptions did not (an example the court provides is "212-819 Hamilton St, Vancouver, B.C. V6Z 6M2 $285,000 - 1 Bed, 1 Bath - Great Newly Updated Jr 1 Bedroom at 819 Hamilton. This is a great West facing 1 bedroom suite which has lots of great. …") copying photographs of properties constituted copyright infringement

Certainly, being a Canadian case (and only a provincial one so far) it's not very persuasive in a US court but I would suggest that a US court probably isn't going to judge things very differently.

Conclusion Purely in my opinion, and I'm certainly not a lawyer and this is not legal advice, I'd suggest that scraping the data in order to produce a very similar version of the same map, especially for commercial purposes, would probably be copyright infringement.

Using the data for research purposes to create a new work, especially if you don't use all of the data, is probably safe under both the fair use and data doctrines.

share|improve this answer
I'm not quite sure why your chastising me. If you publicly publish your data, what do you expect people to do with it? Personally I wouldn't use it for in any direct profitable way (I'm a researcher), and utilizing such data for research purposes wouldn't be any different than using information published in the newspaper. –  Andy W Mar 8 '12 at 18:41
This isn't an answer to my question either. Perhaps bring it up on meta (or another question on the main site) if it really bothers you and you would like other community input. –  Andy W Mar 8 '12 at 18:43
Hmm, I'll admit I'm not a lawyer, and so I could be confused about what is and what is not "copyrightable", but this page suggests that data can not be copyrighted. The Time's does have some guidelines about re-distributing their work, so redistributing the print or web content verbatim would be in some violation I would imagine, but the data itself is not subject to such restrictions. –  Andy W Mar 8 '12 at 21:14
(I meant to question, not chastise, my apologies). Copyright law is very vexed and varies from country to country. Even deciding if something is data or not is a matter of interpretation. For example, Canadian law specifically includes maps in the list of works that can be copyrighted. What you are wanting to scrape above looks like a map (on its face) yet is built from data. So, if it were in Canada would it be copyrighted or not? –  Sarge Mar 8 '12 at 22:45
US law is a bit different to most other jurisdictions because the case law on data includes quite a bit (specifically, the case was about yellow pages type data). Your question comes across as "help me get the data from this site". –  Sarge Mar 8 '12 at 22:47

Your Answer


