GIS in the Browser - The Good Parts
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Priour, Matt
Long gone (hopefully) are the days of replicating the "professionals only" desktop GIS interface in a browser. However, with modern browsers, HTML5 APIs, and increased efficiency of javascript engines it is possible to performantly replicate GIS functionality in a purely client-side browser application. Moderately complex geoprocessing, persistent client-side storage and simple to complex data visualization are all possible now. We walk through the underlying technology and demonstrate the practical use of it in an open-source sample application. Technologies covered include IndexedDB, WebStorage, Workers, Strongly Typed Arrays and Canvas. Some attention will also be paid to performance limitations, browser support and polyfills for older browsers.
 And we'll talk about GIS things we do in the browser these days. And this is clearly not the model that we want anymore. You still see an unfortunately large number of sites like this when you go and look at government and county websites and other places like that that have data or just provide some kind of viewer for information. They've still stuck on a very old model. And a lot of that is also having to do with that so much of that stuff goes back to the server to do anything interesting at all. I mean, most of the stuff is just display. And I think we've moved beyond that. So this is less of a demonstration of what is something that you could just copy and paste and create yourself right now, but more of what's possible and what we should be moving towards. We've got nice modular CSS frameworks and UI frameworks that we don't need a whole solution anymore. You can put together little pieces you want and style it the way you want and make things that are individualized and do things that are specific for your users. So you don't even need to have a web server to look at shape files and that kind of stuff these days. I can't count the number of times that people ask me, well, I've got a shape file and I want to see it on an open layers map or some other kind of map. How do I do that? And for the longest time, I was like, well, you really have to have some kind of server and it's got to be converting that shape file into GeoJSON or something else like that and then serving it out. And so now I've got a nice little drag and drop tools that will let you just drag and drop stuff onto a map like this example from Calvin Metcalf and I can just drop a fairly large number of points onto here and then it sends those back to a worker, it takes the zipped shape file, unzips it and displays the points on your browser and of course, no, I'm doing it live, it's not working or it's working super slow. It was impressive earlier. Maybe we'll come back to that. I might regret doing mostly demos. All right, well, we'll come back to that. So what's powering a lot of this stuff is some newer APIs and they're really not new there but the support for them is now ubiquitous enough that unless you, for some reason, have to support IE8 or less and even with that there's polyfills and stuff like that. So for most of this, IE9 or IE10 will get you and then with all the other modern browsers, the things have been supported for a long time. So being able to draw with Canvas, which provides you a nice drawing surface that you can very quickly draw a lot of information on and it doesn't slow down your browser to SVG support. Again, you see that's IE9 and above and then most powering that other example, which maybe we can get back to. Is Web Workers, which those are a separate process that is in the background of your browser. So it'll spin up extra threads and do stuff on those threads and it doesn't have any real access to the DOM but it does provide a good way to do computation like intensive stuff without slowing down the browser. And the support for it is pretty widespread as long as like I said you're on IE10 or above. There's some variations and differences with those things which is a... You kind of have to deal with but... So like... Is that coming through? No. Let's see if we go... No? Hmm. Alright, well, I'm not able to scale this properly. Okay, well never mind. Anyway, that's the efficiency of passing data back and forth. So things are IE and those other ones are Safari, Firefox and Chrome with the lower being the better. You can do... How long it takes to do a thousand messages. Anyway, so they've got... We've got these things that can help us out and you can do computation like tens of stuff that with the browser still remaining available. Let's see if this one... I'm definitely regretting doing live demos because the three things that I wanted to demonstrate are not working now that I am in front of everybody. So one of the things that makes it easy to use, workers are really gives the workers a good advantage is these strongly typed arrays which again, they're IE10 and above and all the modern browsers. And so it's instead of just a normal JavaScript array which can be a list of any objects, it's like unsigned integers or buffers. So it's numbers. It's just numbers. That can be very quickly passed around by reference or transferred over to different threads and operated on very efficiently. So that kind of stuff really powers some of the new cool stuff you see in WebGL and allows you to do interesting stuff. So WebSQL database, it's never really caught on and that's a shame because it's a good way to store things and gave you a nice, you know, SQLite database in the browser. But we do have WebStorage which is a local storage thing and we do have IndexDB, which is one of these, which is got pretty ubiquitous support. The only thing that's really missing is Safari and that's got a good shim as well. And the next version of Safari which should be coming out very soon now, they made the announcement at Apple thing, is we're going to have full support for it as well. And with that you can use things. So with, you can use stuff like leaflet and if you have to talk to, you know, ArcGIS server or want to use things like that, we've got, there's Esri leaflet which is just another communication layer that lets you talk to, it knows how to talk to the ArcGIS servers and then bring it into a visualization on leaflet which is, gives you a nice interface to use with that. One of the things you can also do is use Terraformer and, I cannot find this stuff. Anyway, that you can use IndexDB and local storage to persist your stuff. So you could pull down a dataset and then it'll get stored in local storage. You could be disconnected and still be able to show it later. So even stuff that's been thought to be fairly complex and that you wanted to do on the server of like taking points and triangulating them into tins and then displaying them, this is a simple example on using D3. But you can use another library called Delaney Fast which you can find and use it directly as stuff like leaflet or open layers or the Esri ArcGIS API and get the same kind of speed and performance which unfortunately I was going to show some of that. But my live demo is not working at all. Well, without my demo I'm kind of done. I'm sorry. Anyway, is there any questions or thoughts or comments? Yes. Yeah, there's size limitations and it depends on the browser and the device. So, you know, if you're like on a desktop versus a mobile device, those are different. Some things like will let you request more space, other things, they will, other browsers will actually make you like destroy the database and do a new one. So, you have to worry about that. Limitations are generally reasonable unless you're storing like lots of data. So, you could get away with a pretty large amount on indexed DB which is significantly more than you could do in local storage. Sure. You said the SQLite web workers that they had some limitations as to what you could actually give them, what data they could work on once you invoke them. Is that simply numerical or is that just primitives that it can work with? No. So, you can pass pretty much anything over to a web worker. The things that you can't pass are DOM elements and the canvas element. So, your native DOM elements, you can't pass those over because there's no DOM over in the browser. I mean, sorry, over in the worker. But anything else you can. So, you can pass it any of the primitives. You can pass it any of these strongly typed arrays and you can pass it blobs. So, one thing you could do is, and that's part of my demo, you take a file so you can have a file input and open a file or just drag it to a drag and drop area. And then you can pass that directly over to a worker and then it can work on it and pass the data back to you. So, that's a, it's nice, it's really nice to have that ability and it makes it so you're not tied to having a server anymore. Just to display and manipulate data to some degree or having to use the desktop application. And if you're going to do, you know, several million geocodes or, you know, you're buffering and coalescing, you know, large geometries and stuff, probably are performance-wise going to do a lot better doing it on a desktop or possibly even server side. So, for moderately complex stuff, simple to moderate complex, the browser has gotten good enough where you can do a lot of interesting stuff in there. Yes? So, I was wondering how that clustering works? So, because this is a client-side library, so you still have to bring all the data over to the client before you do clustering, right? So, you have to reduce the network traffic or... Right. So, there's some strategies for helping out with that and one is to bring that data down and then calculate the clusters over in a worker thread and then send the clustered features. So, the cluster, like, bounding box and its information over back to the main thread. But you're right, it doesn't... It's not... You're not... You're still having to pull that data down. So, if you're doing... You can stream data through web sockets and other things like that. It just depends on what the server is supporting and if you've... So, if you've got as a server that will support streaming, you can also improve your performance that way too. Could you say a bit more about the implications of the Delaney triangulation? I kind of missed how that connected to everything else. So, I was an example of something that's relatively complex and that you could now do efficiently in the browser. Just as like something that even two years ago you wouldn't think about doing without doing it over on a desktop application or saying it server side. So, that's what that was and I had a... I demo had points that were going to be triangulated in shown but unfortunately never did open while we were here. Any other questions? Go ahead. The demo you had which you dropped a file with a SIPT shape file. As I understood, that was supposed to go to workers. Yes. All right. So, is there libraries for unsipping files or do you have to implement that yourself? Yeah, no, there is. There's a shape file JS which understands how to read shape files and then how to unzip them as well. So, you can use that and it's really nice. There's some other interesting things that people do with workers is relate to stuff like that that is just dealing with binary data. It's like creating animated gifs from... There's a thing where you can actually just select a number of gifs from your computer and then it takes those into a web worker, creates an animated gif and gives it back to you. And it's all right there in the browser. So... Okay, so, sorry. Sorry your demo didn't work but the idea of like pushing the zip file out to a web worker and sort of doing that in parallel as the page is loading is really awesome. That's an awesome idea. Do you have any URLs you could share with us for the zip code or the code for doing that so we can look at it offline? Yeah, well, I was... So, I'll put the demo on my GitHub repository and also tweet that information out. So, it's Mprio is my GitHub username, so github.com.mprio and it's Matt Prio on Twitter. So, that first example that I showed, that's actually from Calvin Metcalf and... leaflet.CalvinMetcalf.com and you can drag and drop shape files and topo.json and json onto the map and it'll display it. So, I took that example and then added some other stuff to it from the demo. Yes. If there are no more questions, I just wanted to make a quick announcement. If you were in this track for the whole time, you saw how crowded it was and if you're interested in the same track, track four for after the break, we've decided to swap rooms. Track four and track seven will be swapping rooms because it's a little bigger over there, a little more seating. So, just an FYI. Track seven is around the corner here, 143-144. It's a double room. Thank you.
 And we'll talk about GIS things we do in the browser these days.  And this is clearly not the model that we want anymore.  You still see an unfortunately large number of sites like this when you go and look at  government and county websites and other places like that that have data or just provide  some kind of viewer for information.  They've still stuck on a very old model.  And a lot of that is also having to do with that so much of that stuff goes back to the server to do anything interesting at all.  I mean, most of the stuff is just display.  And I think we've moved beyond that.  So this is less of a demonstration of what is something that you could just copy and paste and create yourself right now,  but more of what's possible and what we should be moving towards.  We've got nice modular CSS frameworks and UI frameworks that we don't need a whole solution anymore.  You can put together little pieces you want and style it the way you want and make things that are individualized  and do things that are specific for your users.  So you don't even need to have a web server to look at shape files and that kind of stuff these days.  I can't count the number of times that people ask me, well, I've got a shape file and I want to see it on an open layers map  or some other kind of map. How do I do that?  And for the longest time, I was like, well, you really have to have some kind of server  and it's got to be converting that shape file into GeoJSON or something else like that and then serving it out.  And so now I've got a nice little drag and drop tools that will let you just drag and drop stuff onto a map  like this example from Calvin Metcalf and I can just drop a fairly large number of points onto here  and then it sends those back to a worker, it takes the zipped shape file, unzips it and displays the points on your browser  and of course, no, I'm doing it live, it's not working or it's working super slow.  It was impressive earlier. Maybe we'll come back to that.  I might regret doing mostly demos.  All right, well, we'll come back to that.  So what's powering a lot of this stuff is some newer APIs and they're really not new there  but the support for them is now ubiquitous enough that unless you, for some reason, have to support IE8 or less  and even with that there's polyfills and stuff like that.  So for most of this, IE9 or IE10 will get you and then with all the other modern browsers,  the things have been supported for a long time. So being able to draw with Canvas,  which provides you a nice drawing surface that you can very quickly draw a lot of information on  and it doesn't slow down your browser to SVG support.  Again, you see that's IE9 and above and then most powering that other example, which maybe we can get back to.  Is Web Workers, which those are a separate process that is in the background of your browser.  So it'll spin up extra threads and do stuff on those threads and it doesn't have any real access to the DOM  but it does provide a good way to do computation like intensive stuff without slowing down the browser.  And the support for it is pretty widespread as long as like I said you're on IE10 or above.  There's some variations and differences with those things which is a...  You kind of have to deal with but...  So like...  Is that coming through? No.  Let's see if we go...  No? Hmm.  Alright, well, I'm not able to scale this properly.  Okay, well never mind.  Anyway, that's the efficiency of passing data back and forth.  So things are IE and those other ones are Safari, Firefox and Chrome with the lower being the better.  You can do...  How long it takes to do a thousand messages.  Anyway, so they've got...  We've got these things that can help us out and you can do computation like tens of stuff that with the browser still remaining available.  Let's see if this one...  I'm definitely regretting doing live demos because the three things that I wanted to demonstrate are not working now that I am in front of everybody.  So one of the things that makes it easy to use,  workers are really gives the workers a good advantage is these strongly typed arrays which again, they're IE10 and above and all the modern browsers.  And so it's instead of just a normal JavaScript array which can be a list of any objects, it's like unsigned integers or buffers.  So it's numbers.  It's just numbers.  That can be very quickly passed around by reference or transferred over to different threads and operated on very efficiently.  So that kind of stuff really powers some of the new cool stuff you see in WebGL and allows you to do interesting stuff.  So WebSQL database, it's never really caught on and that's a shame because it's a good way to store things and gave you a nice, you know, SQLite database in the browser.  But we do have WebStorage which is a local storage thing and we do have IndexDB, which is one of these,  which is got pretty ubiquitous support.  The only thing that's really missing is Safari and that's got a good shim as well.  And the next version of Safari which should be coming out very soon now,  they made the announcement at Apple thing, is we're going to have full support for it as well.  And with that you can use things.  So with, you can use stuff like leaflet and if you have to talk to, you know, ArcGIS server or want to use things like that,  we've got, there's Esri leaflet which is just another communication layer that lets you talk to, it knows how to talk to the ArcGIS servers  and then bring it into a visualization on leaflet which is, gives you a nice interface to use with that.  One of the things you can also do is use Terraformer and, I cannot find this stuff.  Anyway, that you can use IndexDB and local storage to persist your stuff.  So you could pull down a dataset and then it'll get stored in local storage.  You could be disconnected and still be able to show it later.  So even stuff that's been thought to be fairly complex and that you wanted to do on the server of like taking points and triangulating them into tins  and then displaying them, this is a simple example on using D3.  But you can use another library called Delaney Fast which you can find and use it directly as stuff like leaflet or open layers or the Esri ArcGIS API  and get the same kind of speed and performance which unfortunately I was going to show some of that.  But my live demo is not working at all.  Well, without my demo I'm kind of done. I'm sorry.  Anyway, is there any questions or thoughts or comments? Yes.  Yeah, there's size limitations and it depends on the browser and the device.  So, you know, if you're like on a desktop versus a mobile device, those are different.  Some things like will let you request more space, other things, they will, other browsers will actually make you like destroy the database and do a new one.  So, you have to worry about that.  Limitations are generally reasonable unless you're storing like lots of data.  So, you could get away with a pretty large amount on indexed DB which is significantly more than you could do in local storage.  Sure.  You said the SQLite web workers that they had some limitations as to what you could actually give them, what data they could work on once you invoke them.  Is that simply numerical or is that just primitives that it can work with?  No. So, you can pass pretty much anything over to a web worker.  The things that you can't pass are DOM elements and the canvas element.  So, your native DOM elements, you can't pass those over because there's no DOM over in the browser.  I mean, sorry, over in the worker. But anything else you can.  So, you can pass it any of the primitives.  You can pass it any of these strongly typed arrays and you can pass it blobs.  So, one thing you could do is, and that's part of my demo, you take a file so you can have a file input and open a file or just drag it to a drag and drop area.  And then you can pass that directly over to a worker and then it can work on it and pass the data back to you.  So, that's a, it's nice, it's really nice to have that ability and it makes it so you're not tied to having a server anymore.  Just to display and manipulate data to some degree or having to use the desktop application.  And if you're going to do, you know, several million geocodes or, you know, you're buffering and coalescing, you know, large geometries and stuff,  probably are performance-wise going to do a lot better doing it on a desktop or possibly even server side.  So, for moderately complex stuff, simple to moderate complex, the browser has gotten good enough where you can do a lot of interesting stuff in there.  Yes?  So, I was wondering how that clustering works?  So, because this is a client-side library, so you still have to bring all the data over to the client before you do clustering, right?  So, you have to reduce the network traffic or...  Right.  So, there's some strategies for helping out with that and one is to bring that data down and then calculate the clusters over in a worker thread  and then send the clustered features.  So, the cluster, like, bounding box and its information over back to the main thread.  But you're right, it doesn't...  It's not... You're not... You're still having to pull that data down.  So, if you're doing... You can stream data through web sockets and other things like that.  It just depends on what the server is supporting and if you've...  So, if you've got as a server that will support streaming, you can also improve your performance that way too.  Could you say a bit more about the implications of the Delaney triangulation?  I kind of missed how that connected to everything else.  So, I was an example of something that's relatively complex and that you could now do efficiently in the browser.  Just as like something that even two years ago you wouldn't think about doing without doing it over on a desktop application or saying it server side.  So, that's what that was and I had a... I demo had points that were going to be triangulated in shown but unfortunately never did open while we were here.  Any other questions?  Go ahead.  The demo you had which you dropped a file with a SIPT shape file.  As I understood, that was supposed to go to workers.  Yes.  All right. So, is there libraries for unsipping files or do you have to implement that yourself?  Yeah, no, there is. There's a shape file JS which understands how to read shape files and then how to unzip them as well.  So, you can use that and it's really nice.  There's some other interesting things that people do with workers is relate to stuff like that that is just dealing with binary data.  It's like creating animated gifs from...  There's a thing where you can actually just select a number of gifs from your computer and then it takes those into a web worker, creates an animated gif and gives it back to you.  And it's all right there in the browser.  So...  Okay, so, sorry.  Sorry your demo didn't work but the idea of like pushing the zip file out to a web worker and sort of doing that in parallel as the page is loading is really awesome.  That's an awesome idea.  Do you have any URLs you could share with us for the zip code or the code for doing that so we can look at it offline?  Yeah, well, I was...  So, I'll put the demo on my GitHub repository and also tweet that information out.  So, it's Mprio is my GitHub username, so github.com.mprio and it's Matt Prio on Twitter.  So, that first example that I showed, that's actually from Calvin Metcalf and...  leaflet.CalvinMetcalf.com and you can drag and drop shape files and topo.json and json onto the map and it'll display it.  So, I took that example and then added some other stuff to it from the demo.  Yes.  If there are no more questions, I just wanted to make a quick announcement.  If you were in this track for the whole time, you saw how crowded it was and if you're interested in the same track, track four for after the break,  we've decided to swap rooms. Track four and track seven will be swapping rooms because it's a little bigger over there, a little more seating.  So, just an FYI.  Track seven is around the corner here, 143-144. It's a double room.  Thank you.
06_Matt_Priour_720p_0_3647.png 06_Matt_Priour_720p_3648_4667.png 06_Matt_Priour_720p_4668_9851.png 06_Matt_Priour_720p_9852_12971.png 06_Matt_Priour_720p_12972_15167.png 06_Matt_Priour_720p_15168_15287.png 06_Matt_Priour_720p_15288_17375.png 06_Matt_Priour_720p_17376_18251.png 06_Matt_Priour_720p_18252_18323.png 06_Matt_Priour_720p_18324_18431.png 06_Matt_Priour_720p_18432_18887.png 06_Matt_Priour_720p_18888_19631.png 06_Matt_Priour_720p_19632_20471.png 06_Matt_Priour_720p_20472_20831.png 06_Matt_Priour_720p_20832_21443.png 06_Matt_Priour_720p_21444_21527.png 06_Matt_Priour_720p_21528_24275.png 06_Matt_Priour_720p_24276_24323.png 06_Matt_Priour_720p_24324_24947.png 06_Matt_Priour_720p_24948_25019.png 06_Matt_Priour_720p_25020_25103.png 06_Matt_Priour_720p_25104_25535.png 06_Matt_Priour_720p_25536_26207.png 06_Matt_Priour_720p_26208_26591.png 06_Matt_Priour_720p_26592_27071.png 06_Matt_Priour_720p_27072_27371.png 06_Matt_Priour_720p_27372_28151.png 06_Matt_Priour_720p_28152_29099.png 06_Matt_Priour_720p_29100_30071.png 06_Matt_Priour_720p_30072_30767.png 06_Matt_Priour_720p_30768_30851.png 06_Matt_Priour_720p_30852_30917.png