Big (enough) data and strategies for distributed geoprocessing
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kraft, Robin
Big data gets a lot of press these days, but even if you're not geocoding the Twitter firehose, "big enough" data can be a pain - whether you're crashing your database server or simply running out of RAM. Distributed geoprocessing can be even more painful, but for the right job it's a revelation!This session will explore strategies you can use to unlock the power of distributed geoprocessing for the "big enough" datasets that make your life difficult. Granted, geospatial data doesn't always fit cleanly into Hadoop's MapReduce framework. But with a bit of creativity - think in-memory joins, hyper-optimized data schemas, and offloading work to API services or PostGIS - you too can get Hadoop MapReduce working on your geospatial data!Real-world examples will be taken from work on GlobalForestWatch.org, a new platform for exploring and analyzing global data on deforestation. I'll be demoing key concepts using Cascalog, a Clojure wrapper for the Cascading Java library that makes Hadoop and Map/Reduce a lot more palatable. If you prefer Python or Scala, there are wrappers for you too.Hadoop is no silver bullet, but for the right geoprocessing job it's a powerful tool.
 Hi everybody, my name is Robin Kraft. I work at the Data Lab at World Resources Institute. I want to give you caveats before I start talking about big data and all that. I'm not an engineer, so not a software engineer. So, you know, I'm not especially interested in like crazy tuning like the last talk was pretty interesting, but I don't do that. I don't use stuff like GeoMesa, which sounds totally amazing, but you have to think about data patterns on your hard drive in terms of like performance and whatnot. I just need stuff to work quickly enough to get my job done. And I think that's still pretty useful. And WRI is an environmental think tank based in Washington, D.C. They do a lot of policy work in developing countries as well as in the United States. It's a policy and research shop that generates a lot of geospatial data in countries where we operate, but it's not a company like Twitter where, you know, you have massive amounts of data. That said, on occasion we do have pretty substantial amounts of data, and that's what I want to talk about. And sometimes we get into this place where we're between big data and small data or normal data, stuff that you can handle on your laptop, for example, and use your standard tools, ArcGIS, QGIS, whatever. But there's this point where it becomes... Do you have a presentation or just a talk? Oh, shit. I'm sorry. Excuse me. I've been doing this whole thing over here and didn't even know that you guys couldn't see it. I'm sorry. All right. Can you see that now? All right. Okay. Here we go. Here's... Let's see. How do I... All right. I'm sorry. Okay. So here's everything I've said so far. All right. So big enough data is when it's big enough to be a pain in the ass. So, like, I know when I see it, but there's not, like, you know, more than a gigabyte, more than a terabyte, more than a petabyte where you can sort of like draw a line. It's when your tools that you're stand... That you typically use start breaking down. So you run out of RAM on your laptop, your server is crashing, you don't have any disk space, your process is running for weeks or potentially years if you let it run to completion. Stuff just doesn't quite work anymore. And so that's a... That's this point where small data is... You're not in the realm of small data anymore, but the big data toolkit might be overkill. You don't necessarily have time to learn HBase and... Hive and, like, everything in the Hadoop ecosystem and Spark and Cassandra and, you know, all these really amazing tools that most of us don't really need unless you do. But in most cases, I don't. So there's this sort of awkward middle ground where you need to find tools that will support the operations that you need to do, but that aren't necessarily the standard tools that you would have on your laptop. So I'm going to talk about this. This is in the context of globalforestwatch.org, which is an initiative that brings together a bunch of partners to put the best scientific data about forestry and deforestation on the web through, you know, nice web maps and the like. And before I go on, I want to just show you what that looks like. I'm going to be talking about two data sets in particular. This one is a...let's see, so you can see all that, yeah. This one is a Landsat-based global data set that tracks forest loss and gain over the last 12 years. Let's go look at the northern urban over here. This is global 30-meter data generated with Google Earth Engine. So there's some deforestation in pink here over near Mount Hood. I don't know what's going on there, but if we switch to satellite, you can see some things going on. And then there's blue regrowth in various places. So this is a pretty amazing data set generated by University of Maryland and Google, and it's global. So that's the first for anything like this. The other data set that I want to talk about is forest monitoring for action or FORMA. That's the one I've been working on. And that is a modus-based system for tracking deforestation hotspots or we don't like to say deforestation because that's politically charged. But forest loss hotspots, however you want to define deforestation is up to you. But where there are trees and then there are no more, that's what we want to identify. So I'm just going to zoom in here to Indonesia, one of the major hotspots. And what's interesting about FORMA is that it's updated every 16 days. So you can see the sort of viral spread of deforestation across the landscape as you can hopefully see here. And the idea is that we want people on the ground to be able to react quickly to forest loss as quickly as possible. And, you know, this is, for forestry, this is considered near real time. In the past, for a country like Indonesia, you'd get a new map of deforestation every couple of years. So, you know, with the 30-meter annual Landsat-based data from University of Maryland and our 500-meter 16-day resolution dataset, you have some pretty cool new tools built into Global Forest Watch that you can use for if you're doing stuff with international forestry. So back to the presentation. Why is this not full screen? All right, here we go. Okay. Done with demo. So now I'm going to talk a little bit about the nuts and bolts of how FORMA works. But first, I don't know how many of you saw the interesting talk yesterday by the guy that does leaflet. He's talking about how simplicity was a guiding, what is the goal of, should be the goal, one of the goals. And, you know, I think that he made a lot of good points in his talk. But one in particular that I heard actually at another conference was that simplicity in some cases is better than optimal because basically an hour of human time, I heard this recently, if you can, with an hour of one of our salaries, you can buy 400 hours on Amazon to crunch whatever stuff you're crunching. So there's a real question about how much time you want to put into optimizing the hell out of a process when if your process can just scale, you can save a lot of money and time that way instead. And I know that this is not going to work in every case, but it's something to think about, something to keep in mind. So FORMA is basically an image processing algorithm. It takes in a lot of satellite imagery from NASA, the MODIS vegetation index dataset. And then we do statistics. Basically what you see here is one pixel with the vegetation intensity shown over time. So this is the NDVI. It's basically a measure of vegetation intensity, your greenness, and it has seasonal fluctuations as even in the tropics where there's, there aren't seasons as we would, as we would recognize them here. But the important thing is that the NDVI, even with the seasonal fluctuations and, you know, cloud cover and whatnot, it does have a pretty seasonal, pretty seasonal, pre-predictable behavior. So if you have something that goes from green and intense vegetation to brown or not very intense vegetation, and you happen to have fires around the same time, which you see at the end of that time series, that is something that might be considered deforestation. How exactly we classify as deforestation depends on a model built around historical deforestation. So we're looking for patterns in the NDVI signal that are indicative of deforestation. But the point is for the purposes of this talk is that we have these pixel, we need to build pixel time series so that we can run regressions on them. We need to do spatial joins so that we can bring in other data sets like rainfall or fires, which are not in the same format as the raw NDVI that we're using. We need spatial filters because we're not actually, we don't care about deforestation over land, right? So, over the ocean, right? So we need to filter that out and we need to be able to do statistics, just, you know, standard statistics that you would want to do in, like, econometrics that are not necessarily designed for working with images. So when we first started doing this, we were using one or two desktop machines. They were just both hitting the same hard drive. We didn't really know what we were doing at that point. And we were using ArcGIS and Python using some data in NumPy to do the actual math. And it worked, but that was for a very small number of pixels, just to show that the algorithm had legs and that it worked in Brazil and Indonesia, just on these little postage stamps. So we then have struggled, we struggled subsequently on how to scale that up from 10,000 pixels, 100 kilometers squared, one kilometer resolution to 100 billion pixels at 500 meter resolution covering the whole tropics. So the sort of insight that we got was that if we treat everything as a raster, that helps us in certain ways and causes problems in others. But I'm actually, unlike this guy, I'm a fan of rasters. I think they're really amazing data type. And if you treat everything as a raster, you can treat everything as text because at the end of the day, you can convert a raster into a text, you know, this very simple pick raster here, change it into row columns and values and you have a, you have something you can throw into a database or just write as text files. And since everything is also a raster, you can, since everything is a raster at the end of the day, you can convert anything into, anything that we care about into rasters, whether it's points, polygons, lines and whatnot. You can convert those same pieces of data into text and that's great because Hadoop loves text and that's where we get to the biggest data questions here. So the problem here is that Hadoop is not simple. I don't know how many of you are familiar with writing map readers jobs, but it's not a very intuitive way of thinking about how to process data, especially geospatial data. So what we ended up doing is using a technology stack that uses closure, cascading and cascalog to basically take away a lot of the pain of working with Hadoop. We also run this on elastic map reduce on Amazon, so that's also convenient. But closure is nice because it's a cool, very nice language to work with. It's a Lisp. It's very elegant and if you're in the Lisp, you'll appreciate closure. If you don't know closure or Lisp, it's just a, you'll see a little bit of it today and it's weird, but it's good. Cascading is a really cool library that basically writes map reduce jobs for you. So you give it, you give it, you basically tell it what to do and it will write the map reduce jobs for you, run them on your Hadoop stack. And cascalog is just a closure wrapper for that library. So you get map, the benefits of map reduce, which is a lot of scalability if you can express your problem in terms of map reduce, but you don't have to think about map reduce. So here I just want to do a little bit of basic code walkthrough. It looks like that's kind of tiny. I don't know if I can, that might be a little better. So all this is doing is taking in a data source which has row columns and values, multiplying the value by five on that last line there, and then spitting out the results. So the row column and the new value. It's, this can run just on your laptop. It can also run on a massive cluster. And you see at the bottom what you get. One of the other interesting things about, one interesting thing about, interesting thing about cascalog is that you can do these implicit joins. So in this case we have a pixel source, you know, just the row column and value. And we have a data set that represents the countries that those pixels fall into, which we've generated previously somehow. So basically by giving, by taking a pixel source and a country source and naming the fields the same, so the row and column are named the same in each source, you can see, do you see my pointer? Yeah. So like right here, pixel source, row column, row column there. We can do an implicit join by specifying row column in the output vector here. And there we've done what amounts to a spatial join in three lines of code on your laptop or on a hundred servers. This is getting more complicated, but not really, because all we're trying to do is join fires, which happen at a certain latitude and longitude with a country and trying to count up how many fires happen in each country. So again, you've got, we've got a fire source here that has a latitude and longitude, the date and the brightness. We've got the country source, which is in rows and columns again. Sorry guys. We have a function that just converts from latitude and longitude to rows and columns. We filter on the brightness because we only want hot fires. And then we count up how many fires happen in a particular country because the country is what's in the output vector. And so it does an implicit join again to give us the result at the bottom where Indonesia is a hot spot of fires. Here we have a pixel time, we're trying to build a pixel time series. So that's essential for doing our regression analysis. And so, you know, you've got, you know, you can imagine this being having originally been four different rasters, two pixels and four different rasters over time. All we have to do to have this scale from one laptop to a hundred servers is have a function called build series that takes in the date and the value and spits out a time series, which, you know, is just a vector of values. So this is what we get at the end. Just a nice, nice clean vector of values. If we're using this subsequently, we can then, we can use this, these values to, to like pass a regression line through and see if the, see if the values are statistically significantly changing over time. Just standard, standard stuff. So that's, that's how we, that's how we do a lot of the data manipulations in the, in the actual form of algorithm. The 90% of the code is actually just moving data around, doing the joins, doing the, bringing in different data sources, making sure everything lines up correctly. But then, and then, and then, you know, we had to, we have to do just standard statistics. So there's, there are software libraries that take care of that for us. But then, once we have our data set of all the deforestation that, that we've detected, we need to put that onto the map that I showed you. So the guys at Visuality developed the site for us and they came up with this crazy data type, which is sort of like vector tiles, except it's just text, not some binary, binary format that Mapbox is working on. But basically, you have these X and Y, X and Y fields that tell your browser where to paint a pixel on the, on the screen. And so you get these really smooth, the really smooth animation that you, that you saw on the, in the demo. So that's just doing one, it's not swapping out tiles at all, because that's really inefficient, it's just redrawing, redrawing pixels as time moves forward. So this sequel here is what we were using to generate those, the different zoom levels. Andrew Hill at Visual, at CartierDb, wrote this code. I didn't have to think about this, which is really nice. But the problem was that it got really slow and sometimes we're getting server timeouts for a large table, it just was not efficient. So I was looking for, unfortunately we launched using these and this was, I was at, I was up at four in the morning on launch day trying to make sure that these sequel queries finished because, you know, we just kept updating the data. We had to launch and the sequel queries kept failing because we were also, you know, we were getting launch traffic and anyway, it was turning into a nightmare. So what we do now instead, because it was hard to test, it kept breaking, is we use Cascalog to generate the values that go into the table. So we have this very simple calculation that takes a, takes an XYZ coordinate, calculates the values at different zoom levels and the updates the XYZ values. This is the Cascalog query that actually does this over that, or no, this actually generates the XYZ coordinates from lat long or from row and column going through lat long into XYZ. And then you, as a result, you are transposing the time series and into long format which we can then use to count up how many deforestation events happened in a particular area. And that's how we then paint the change in the forest cover on the website. And then the query that actually generates all those zoom levels is just this three line, this three line thing. This is just telling us where the data comes from, generate the tiles using the function above, count up how many deforestation events happen in each of those tiles. And what's nice is that, again, that scales from your laptop to 100 servers pretty simply. So the nice thing about that is instead of having to babysit these SQL queries hoping that they finish, hoping that the server is not under too much load to handle the update, instead we get something that's basically infinitely scalable. We can test every bit of the code before we deploy it in production. It's a very reliable process and it's fast enough. So we're not getting, this process might, you know, instead of taking, instead of having something super optimized that would take a few minutes, this might take an hour or if I throw more machines at it, it'll take 15 minutes. But the idea is that we don't have to always optimize everything down to the last millisecond. And if you can horizontally scale your process, you can just throw more machines at it until you get it quick enough for your purposes. So just to wrap up here, the lessons for big enough geospatial data is the first one, echoing Mike Bostock yesterday, his great talk, was find the right tools and actually use them. Don't get stuck using the same old tools that you've used in the past that aren't quite the right things for what you're trying to do now. There are a lot of great tools out there for doing process, distributed processing. This is just a sampling of them. I've used Hadoop's Star Cluster, Spark. Actually, I guess I've used all of these except for GeoTrailers. But depending on your use case and your application, each of these can have a role in processing data sets that you wouldn't otherwise really be able to handle in your normal tool set. It's useful to keep in mind that simple is or can be better than optimal. You're very, very expensive. If you can get computers to do your work, you're saving yourself money and time. If you're creative about data formats, if you're not worried so much about, you know, using strict geospatial data types and indices and stuff, if you're creative about what geospatial means versus just pure text, you can explore some tools that otherwise would be unavailable to you. The last thing is that Hadoop can be really great, but it's very powerful, but it can also be really painful. So keeping things simple with a library like Cascading that will do the work of MapReduce for you is a really nice thing. So that's that. My name's Robin. Any questions? I got a question. What's up? Oh, sorry. So closure is really cool. How are you distributing and managing? So like you actually want to, your data sets are really big, so now you need to scale out to 100 nodes on AWS. What are you actually using to manage that job and then distribute the data set? So we actually work completely on AWS, so we don't ever have the data locally, so we just keep it on S3, and natively it's available to our Hadoop system, so it's just all there, and it's available to every node. So philosophically, I'm trying to wrap my head around the raster to text expression in terms of thinking about your spatial data as a text. In terms of how do you assign a persistent ID to raster to figure out what part of the world that little piece of text talks about, could you just give me 30 seconds more of that? Sure, that's a great question. So, modus data are split up into tiles, so there are 10 degrees across, so you can, based on the, like a given latitude and longitude can be converted into a tile coordinate, and to tell you, you can figure out which tile it falls into and then which pixel within that tile it falls into. So we have this mapper from lat long to tile and image coordinates, and that's, so we can go back and forth. And that's persistent for the modus? Yes. This library? Yeah, the modus is great for just how incredibly consistent and systematic it is. Got it. So that's super helpful for us. So how was the data broken down when you're processing it? Did you process it all, or, I don't know too much about the Hadoop cluster, but does it process it a chunk at a time? Yeah, so if, so you can, so at the beginning we have to start out by processing each file, each image file individually, because you can't split those natively in Hadoop. So we just read in one file, basically split it up into chunks and spit it back out. And then those just go into text files, which, well, sequence files, which is Hadoop's binary format for storing data. And we just, it just spits out rows and rows and rows of values and Hadoop handles all the splitting for us. So we never really think about that again. Got it. Yeah. We started playing with this quite a bit too, and I still have to wrap my brand about, you know, that geographic text conversion. Is, will that always be the way it is? Will people write geographic wrappers around such that Hadoop can, not Hadoop specifically, but something on top of Hadoop can take geographic data more natively? Yeah, that's a great question. There was a pretty cool talk yesterday about a project called GeoMesa that does just that. And it sounds like a very high performance way to do geospatial natively in the Hadoop context. The more you get into trying to do, like, real geospatial, the more complex it gets. And you have to ask whether the return on the investment is worth it. And for us, it wasn't. But then again, we're also not computer scientists. So, you know, learning, like the learning curve, like, it was bad enough just having to get up this learning curve, going that much further was just, it was just too much. But there are people that are working on that. I'm hoping that there will be a tool set, a geospatial tool set that's just sort of like works out of the box or like Postus is today. And, well, out of the box-ish. Like Postus is today. But I haven't come across that yet. But I'd love to know if somebody else has. All right. Thank you.
 Hi everybody, my name is Robin Kraft. I work at the Data Lab at World Resources Institute.  I want to give you caveats before I start talking about big data and all that. I'm  not an engineer, so not a software engineer. So, you know, I'm not especially interested  in like crazy tuning like the last talk was pretty interesting, but I don't do that.  I don't use stuff like GeoMesa, which sounds totally amazing, but you have to think about  data patterns on your hard drive in terms of like performance and whatnot. I just need  stuff to work quickly enough to get my job done. And I think that's still pretty useful.  And WRI is an environmental think tank based in Washington, D.C. They do a lot of policy  work in developing countries as well as in the United States. It's a policy and research  shop that generates a lot of geospatial data in countries where we operate, but it's not  a company like Twitter where, you know, you have massive amounts of data. That said, on  occasion we do have pretty substantial amounts of data, and that's what I want to talk about.  And sometimes we get into this place where we're between big data and small data or normal  data, stuff that you can handle on your laptop, for example, and use your standard tools,  ArcGIS, QGIS, whatever. But there's this point where it becomes...  Do you have a presentation or just a talk?  Oh, shit. I'm sorry. Excuse me. I've been doing this whole thing over here and didn't  even know that you guys couldn't see it. I'm sorry. All right. Can you see that now? All  right. Okay. Here we go. Here's... Let's see. How do I... All right. I'm sorry. Okay. So  here's everything I've said so far. All right. So big enough data is when it's big enough  to be a pain in the ass. So, like, I know when I see it, but there's not, like, you know,  more than a gigabyte, more than a terabyte, more than a petabyte where you can sort of  like draw a line. It's when your tools that you're stand... That you typically use start  breaking down. So you run out of RAM on your laptop, your server is crashing, you don't  have any disk space, your process is running for weeks or potentially years if you let  it run to completion. Stuff just doesn't quite work anymore. And so that's a... That's  this point where small data is... You're not in the realm of small data anymore, but the  big data toolkit might be overkill. You don't necessarily have time to learn HBase and...  Hive and, like, everything in the Hadoop ecosystem and Spark and Cassandra and, you know, all  these really amazing tools that most of us don't really need unless you do. But in most  cases, I don't. So there's this sort of awkward middle ground where you need to find tools  that will support the operations that you need to do, but that aren't necessarily the  standard tools that you would have on your laptop. So I'm going to talk about this.  This is in the context of globalforestwatch.org, which is an initiative that brings together  a bunch of partners to put the best scientific data about forestry and deforestation on the  web through, you know, nice web maps and the like. And before I go on, I want to just show  you what that looks like. I'm going to be talking about two data sets in particular. This one  is a...let's see, so you can see all that, yeah. This one is a Landsat-based global data  set that tracks forest loss and gain over the last 12 years. Let's go look at the northern  urban over here. This is global 30-meter data generated with Google Earth Engine. So there's  some deforestation in pink here over near Mount Hood. I don't know what's going on there,  but if we switch to satellite, you can see some things going on. And then there's blue  regrowth in various places. So this is a pretty amazing data set generated by University of  Maryland and Google, and it's global. So that's the first for anything like this. The other  data set that I want to talk about is forest monitoring for action or FORMA. That's the  one I've been working on. And that is a modus-based system for tracking deforestation hotspots  or we don't like to say deforestation because that's politically charged. But forest loss  hotspots, however you want to define deforestation is up to you. But where there are trees and  then there are no more, that's what we want to identify. So I'm just going to zoom in  here to Indonesia, one of the major hotspots. And what's interesting about FORMA is that  it's updated every 16 days. So you can see the sort of viral spread of deforestation across  the landscape as you can hopefully see here. And the idea is that we want people on the  ground to be able to react quickly to forest loss as quickly as possible. And, you know,  this is, for forestry, this is considered near real time. In the past, for a country  like Indonesia, you'd get a new map of deforestation every couple of years. So, you know, with the  30-meter annual Landsat-based data from University of Maryland and our 500-meter 16-day resolution  dataset, you have some pretty cool new tools built into Global Forest Watch that you can  use for if you're doing stuff with international forestry. So back to the presentation. Why  is this not full screen? All right, here we go. Okay. Done with demo. So now I'm going  to talk a little bit about the nuts and bolts of how FORMA works. But first, I don't know  how many of you saw the interesting talk yesterday by the guy that does leaflet. He's talking  about how simplicity was a guiding, what is the goal of, should be the goal, one of the  goals. And, you know, I think that he made a lot of good points in his talk. But one  in particular that I heard actually at another conference was that simplicity in some cases  is better than optimal because basically an hour of human time, I heard this recently,  if you can, with an hour of one of our salaries, you can buy 400 hours on Amazon to crunch  whatever stuff you're crunching. So there's a real question about how much time you want  to put into optimizing the hell out of a process when if your process can just scale, you can  save a lot of money and time that way instead. And I know that this is not going to work  in every case, but it's something to think about, something to keep in mind. So FORMA  is basically an image processing algorithm. It takes in a lot of satellite imagery from  NASA, the MODIS vegetation index dataset. And then we do statistics. Basically what  you see here is one pixel with the vegetation intensity shown over time. So this is the  NDVI. It's basically a measure of vegetation intensity, your greenness, and it has seasonal  fluctuations as even in the tropics where there's, there aren't seasons as we would,  as we would recognize them here. But the important thing is that the NDVI, even with  the seasonal fluctuations and, you know, cloud cover and whatnot, it does have a pretty  seasonal, pretty seasonal, pre-predictable behavior. So if you have something that goes  from green and intense vegetation to brown or not very intense vegetation, and you happen  to have fires around the same time, which you see at the end of that time series, that  is something that might be considered deforestation. How exactly we classify as deforestation depends  on a model built around historical deforestation. So we're looking for patterns in the NDVI  signal that are indicative of deforestation. But the point is for the purposes of this  talk is that we have these pixel, we need to build pixel time series so that we can run  regressions on them. We need to do spatial joins so that we can bring in other data sets  like rainfall or fires, which are not in the same format as the raw NDVI that we're using.  We need spatial filters because we're not actually, we don't care about deforestation  over land, right? So, over the ocean, right? So we need to filter that out and we need  to be able to do statistics, just, you know, standard statistics that you would want to  do in, like, econometrics that are not necessarily designed for working with images. So when  we first started doing this, we were using one or two desktop machines. They were just  both hitting the same hard drive. We didn't really know what we were doing at that point.  And we were using ArcGIS and Python using some data in NumPy to do the actual math.  And it worked, but that was for a very small number of pixels, just to show that the algorithm  had legs and that it worked in Brazil and Indonesia, just on these little postage stamps.  So we then have struggled, we struggled subsequently on how to scale that up from 10,000 pixels,  100 kilometers squared, one kilometer resolution to 100 billion pixels at 500 meter resolution  covering the whole tropics. So the sort of insight that we got was that if we treat everything  as a raster, that helps us in certain ways and causes problems in others. But I'm actually,  unlike this guy, I'm a fan of rasters. I think they're really amazing data type. And if you  treat everything as a raster, you can treat everything as text because at the end of the  day, you can convert a raster into a text, you know, this very simple pick raster here,  change it into row columns and values and you have a, you have something you can throw  into a database or just write as text files. And since everything is also a raster, you  can, since everything is a raster at the end of the day, you can convert anything into,  anything that we care about into rasters, whether it's points, polygons, lines and whatnot.  You can convert those same pieces of data into text and that's great because Hadoop loves  text and that's where we get to the biggest data questions here. So the problem here is  that Hadoop is not simple. I don't know how many of you are familiar with writing map  readers jobs, but it's not a very intuitive way of thinking about how to process data,  especially geospatial data. So what we ended up doing is using a technology stack that  uses closure, cascading and cascalog to basically take away a lot of the pain of working with  Hadoop. We also run this on elastic map reduce on Amazon, so that's also convenient. But  closure is nice because it's a cool, very nice language to work with. It's a Lisp. It's  very elegant and if you're in the Lisp, you'll appreciate closure. If you don't know closure  or Lisp, it's just a, you'll see a little bit of it today and it's weird, but it's good.  Cascading is a really cool library that basically writes map reduce jobs for you. So you give  it, you give it, you basically tell it what to do and it will write the map reduce jobs  for you, run them on your Hadoop stack. And cascalog is just a closure wrapper for that  library. So you get map, the benefits of map reduce, which is a lot of scalability if you  can express your problem in terms of map reduce, but you don't have to think about map reduce.  So here I just want to do a little bit of basic code walkthrough. It looks like that's  kind of tiny. I don't know if I can, that might be a little better. So all this is doing  is taking in a data source which has row columns and values, multiplying the value by five  on that last line there, and then spitting out the results. So the row column and the  new value. It's, this can run just on your laptop. It can also run on a massive cluster.  And you see at the bottom what you get. One of the other interesting things about, one  interesting thing about, interesting thing about cascalog is that you can do these implicit  joins. So in this case we have a pixel source, you know, just the row column and value. And  we have a data set that represents the countries that those pixels fall into, which we've generated  previously somehow. So basically by giving, by taking a pixel source and a country source  and naming the fields the same, so the row and column are named the same in each source,  you can see, do you see my pointer? Yeah. So like right here, pixel source, row column,  row column there. We can do an implicit join by specifying row column in the output vector  here. And there we've done what amounts to a spatial join in three lines of code on your  laptop or on a hundred servers. This is getting more complicated, but not really, because  all we're trying to do is join fires, which happen at a certain latitude and longitude  with a country and trying to count up how many fires happen in each country. So again,  you've got, we've got a fire source here that has a latitude and longitude, the date and  the brightness. We've got the country source, which is in rows and columns again. Sorry  guys. We have a function that just converts from latitude and longitude to rows and columns.  We filter on the brightness because we only want hot fires. And then we count up how many  fires happen in a particular country because the country is what's in the output vector.  And so it does an implicit join again to give us the result at the bottom where Indonesia  is a hot spot of fires. Here we have a pixel time, we're trying to build a pixel time series.  So that's essential for doing our regression analysis. And so, you know, you've got, you  know, you can imagine this being having originally been four different rasters, two pixels and  four different rasters over time. All we have to do to have this scale from one laptop to  a hundred servers is have a function called build series that takes in the date and the  value and spits out a time series, which, you know, is just a vector of values. So this  is what we get at the end. Just a nice, nice clean vector of values. If we're using this  subsequently, we can then, we can use this, these values to, to like pass a regression  line through and see if the, see if the values are statistically significantly changing over  time. Just standard, standard stuff. So that's, that's how we, that's how we do a lot of the  data manipulations in the, in the actual form of algorithm. The 90% of the code is actually  just moving data around, doing the joins, doing the, bringing in different data sources,  making sure everything lines up correctly. But then, and then, and then, you know, we  had to, we have to do just standard statistics. So there's, there are software libraries that  take care of that for us. But then, once we have our data set of all the deforestation  that, that we've detected, we need to put that onto the map that I showed you. So the  guys at Visuality developed the site for us and they came up with this crazy data type,  which is sort of like vector tiles, except it's just text, not some binary, binary format  that Mapbox is working on. But basically, you have these X and Y, X and Y fields that  tell your browser where to paint a pixel on the, on the screen. And so you get these really  smooth, the really smooth animation that you, that you saw on the, in the demo. So that's  just doing one, it's not swapping out tiles at all, because that's really inefficient,  it's just redrawing, redrawing pixels as time moves forward. So this sequel here is what  we were using to generate those, the different zoom levels. Andrew Hill at Visual, at CartierDb,  wrote this code. I didn't have to think about this, which is really nice. But the problem  was that it got really slow and sometimes we're getting server timeouts for a large  table, it just was not efficient. So I was looking for, unfortunately we launched using  these and this was, I was at, I was up at four in the morning on launch day trying to make  sure that these sequel queries finished because, you know, we just kept updating the data.  We had to launch and the sequel queries kept failing because we were also, you know, we  were getting launch traffic and anyway, it was turning into a nightmare. So what we do  now instead, because it was hard to test, it kept breaking, is we use Cascalog to generate  the values that go into the table. So we have this very simple calculation that takes a,  takes an XYZ coordinate, calculates the values at different zoom levels and the updates the  XYZ values. This is the Cascalog query that actually does this over that, or no, this  actually generates the XYZ coordinates from lat long or from row and column going through  lat long into XYZ. And then you, as a result, you are transposing the time series and into  long format which we can then use to count up how many deforestation events happened  in a particular area. And that's how we then paint the change in the forest cover on the  website. And then the query that actually generates all those zoom levels is just this  three line, this three line thing. This is just telling us where the data comes from,  generate the tiles using the function above, count up how many deforestation events happen  in each of those tiles. And what's nice is that, again, that scales from your laptop  to 100 servers pretty simply. So the nice thing about that is instead of having to babysit  these SQL queries hoping that they finish, hoping that the server is not under too much  load to handle the update, instead we get something that's basically infinitely scalable.  We can test every bit of the code before we deploy it in production. It's a very reliable  process and it's fast enough. So we're not getting, this process might, you know, instead  of taking, instead of having something super optimized that would take a few minutes, this  might take an hour or if I throw more machines at it, it'll take 15 minutes. But the idea  is that we don't have to always optimize everything down to the last millisecond. And if you can  horizontally scale your process, you can just throw more machines at it until you get it  quick enough for your purposes. So just to wrap up here, the lessons for big enough geospatial  data is the first one, echoing Mike Bostock yesterday, his great talk, was find the right  tools and actually use them. Don't get stuck using the same old tools that you've used  in the past that aren't quite the right things for what you're trying to do now. There are  a lot of great tools out there for doing process, distributed processing. This is just a sampling  of them. I've used Hadoop's Star Cluster, Spark. Actually, I guess I've used all of  these except for GeoTrailers. But depending on your use case and your application, each  of these can have a role in processing data sets that you wouldn't otherwise really be  able to handle in your normal tool set. It's useful to keep in mind that simple is or can  be better than optimal. You're very, very expensive. If you can get computers to do  your work, you're saving yourself money and time. If you're creative about data formats,  if you're not worried so much about, you know, using strict geospatial data types and indices  and stuff, if you're creative about what geospatial means versus just pure text, you can explore  some tools that otherwise would be unavailable to you. The last thing is that Hadoop can  be really great, but it's very powerful, but it can also be really painful. So keeping  things simple with a library like Cascading that will do the work of MapReduce for you  is a really nice thing. So that's that. My name's Robin. Any questions?  I got a question. What's up? Oh, sorry. So closure is really cool. How are you distributing  and managing? So like you actually want to, your data sets are really big, so now you  need to scale out to 100 nodes on AWS. What are you actually using to manage that job  and then distribute the data set? So we actually work completely on AWS, so we don't ever  have the data locally, so we just keep it on S3, and natively it's available to our Hadoop  system, so it's just all there, and it's available to every node.  So philosophically, I'm trying to wrap my head around the raster to text expression  in terms of thinking about your spatial data as a text. In terms of how do you assign a  persistent ID to raster to figure out what part of the world that little piece of text  talks about, could you just give me 30 seconds more of that?  Sure, that's a great question. So, modus data are split up into tiles, so there are 10  degrees across, so you can, based on the, like a given latitude and longitude can be  converted into a tile coordinate, and to tell you, you can figure out which tile it  falls into and then which pixel within that tile it falls into. So we have this mapper  from lat long to tile and image coordinates, and that's, so we can go back and forth.  And that's persistent for the modus?  Yes.  This library?  Yeah, the modus is great for just how incredibly consistent and systematic it is.  Got it.  So that's super helpful for us.  So how was the data broken down when you're processing it? Did you process it all, or,  I don't know too much about the Hadoop cluster, but does it process it a chunk at a time?  Yeah, so if, so you can, so at the beginning we have to start out by processing each file,  each image file individually, because you can't split those natively in Hadoop. So we  just read in one file, basically split it up into chunks and spit it back out. And then  those just go into text files, which, well, sequence files, which is Hadoop's binary format  for storing data. And we just, it just spits out rows and rows and rows of values and Hadoop  handles all the splitting for us. So we never really think about that again.  Got it.  Yeah.  We started playing with this quite a bit too, and I still have to wrap my brand about,  you know, that geographic text conversion. Is, will that always be the way it is? Will  people write geographic wrappers around such that Hadoop can, not Hadoop specifically,  but something on top of Hadoop can take geographic data more natively?  Yeah, that's a great question. There was a pretty cool talk yesterday about a project  called GeoMesa that does just that. And it sounds like a very high performance way to  do geospatial natively in the Hadoop context. The more you get into trying to do, like,  real geospatial, the more complex it gets. And you have to ask whether the return on  the investment is worth it. And for us, it wasn't. But then again, we're also not computer  scientists. So, you know, learning, like the learning curve, like, it was bad enough just  having to get up this learning curve, going that much further was just, it was just too  much. But there are people that are working on that. I'm hoping that there will be a tool  set, a geospatial tool set that's just sort of like works out of the box or like Postus  is today. And, well, out of the box-ish. Like Postus is today. But I haven't come across  that yet. But I'd love to know if somebody else has.  All right. Thank you.
03_Robin_Kraft_720p_0_2423.png 03_Robin_Kraft_720p_2424_2543.png 03_Robin_Kraft_720p_2544_2795.png 03_Robin_Kraft_720p_2796_2975.png 03_Robin_Kraft_720p_2976_4487.png 03_Robin_Kraft_720p_4488_4787.png 03_Robin_Kraft_720p_4788_8975.png 03_Robin_Kraft_720p_8976_9671.png 03_Robin_Kraft_720p_9672_11015.png 03_Robin_Kraft_720p_11016_14735.png 03_Robin_Kraft_720p_14736_15263.png 03_Robin_Kraft_720p_15264_18371.png 03_Robin_Kraft_720p_18372_19271.png 03_Robin_Kraft_720p_19272_26975.png 03_Robin_Kraft_720p_26976_29099.png 03_Robin_Kraft_720p_29100_32603.png 03_Robin_Kraft_720p_32604_33059.png 03_Robin_Kraft_720p_33060_33779.png 03_Robin_Kraft_720p_33780_35279.png 03_Robin_Kraft_720p_35280_36167.png 03_Robin_Kraft_720p_36168_36263.png 03_Robin_Kraft_720p_36264_36311.png 03_Robin_Kraft_720p_36312_37583.png 03_Robin_Kraft_720p_37584_37631.png 03_Robin_Kraft_720p_37632_37679.png 03_Robin_Kraft_720p_37680_39901.png