The role of geospatial open source (FOSS4G) as a component of hybrid systems
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Terner, Michael
Currently, it is rare that an organization faces a choice between "just commercial" or "just open source" geospatial software. Increasingly, the answer isn't about "or", it is now about solutions that may involve both. Indeed, Commercial systems - both installed software and cloud-based platforms - may have both functional and cost effectiveness gaps. Free and open source software for geospatial (FOSS4G) can be a critical component that helps to bridge these gaps. And, given that organizations may have large existing investments and sunk costs in commercial software it can make sense to leverage those investments by building hybrid systems that incorporate open source components. This presentation will describe several specific examples of hybrid systems that incorporate geospatial open source (e.g., OpenGeo Suite) alongside both Esri and Google components. The presentation will focus on both the business reasoning for choosing hybrid as well as the technical approaches taken. Ultimately, hybrid projects or systems can be very important as they provide a key entry-point for geospatial open source into organizations that might not otherwise consider it.
 Good morning. My name is Michael Turner. I'm one of the partners in applied geographic. And I guess in the spirit of free and open source, it's open moderation here. So I'm going to just dive in and talk for about 20 minutes on hybrid systems. I think the place to start is with the definition of hybrid, at least my definition of hybrid. And essentially, any solution that requires more than one piece of technology, one flavor of technology, one stack of technology, and those technologies may be open or commercial or both. And I really like this quote from a blog that Esri did a few years back. Those of you who have seen a recent blog posting, I did. I quoted this as well. But the real point is that the technology doesn't matter. What really matters is what are you trying to do, and you're trying to pick the best tools for getting it done. And it doesn't all have to be one tool. They can be intermixed. I really appreciate that aesthetic. Another way of thinking about it is it's best of breed. Don't be afraid to mix and match things because some things are very good for one thing or the other thing. You can have the best geocoder and the best map publishing environment and the best desktop environment if that's part of your solution. Just as with dogs and where best of breed comes from, some dogs are good in the water, some dogs are good hunting, some dogs are good herding. You get the one that you need. But it's a big marketplace. Our geospatial environment is a big marketplace. And it's hard or can be difficult to make choices when there are so many choices, you know, and so many pieces of technology, whether it's desktop or JavaScript frameworks or geospatial servers or even database servers. And then what's best commercial, what's best open source. So it's not a simple task to sort of implement best of breed. You have to be a good consumer or a smart shopper, if you will. I had this slide this morning after seeing Paul Ramsey's talk on post-just for managers. I think it's a real important set of observations that it's not an all or nothing that sometimes starting with a small constrained open source component can get the ball rolling. And what he was saying in the context of post-just but substitute any geospatial open source is maybe you have two things for a little while. Your staff get used to it. They build confidence and then bit by bit you migrate or build a dual capacity. And you know, it's important to also, you know, respect where commercial software came from. Just because open source is readily available, readily downloadable, the savings aren't immediately there. Most commercial software is paid for, was paid for a long time ago. And there are real business costs to switching. And economic term, a switching cost can be non-trivial if you have a big existing mature operation. And again, you know, a few more details on the commercial side. You know, we've been around 23 plus years and, you know, I grew up in an environment in the, in the night, you know, professionally grew up in an environment in the 1980s where there weren't geospatial open source alternatives. So a lot of people have commercial stuff because they started a long time ago. And it's not just the costs and money. Organizations have a lot of expertise and staff knowledge about how the commercial systems work. And in my opinion, there are, you know, some of the commercial systems remain the best of greed. If you want to do real high-end cartography, if you want to do sort of esoteric geospatial analysis, et cetera, you might judge that commercial systems are very, very competitive. And obviously, one of the things that's starting to emerge in the open source system, ecosystem, is a support network. Places to go for help, places where you can buy support and things of that nature. And again, in commercial, that's always been there. That's one of the things you very much are buying. So that's sort of a view of some of the things that Esri really does well and the core of their business. And then I'm going to talk about some Esri stuff and some Google stuff. And on Google, what people are looking for is the scalability of their cloud, the ability to have applications that can support hundreds of thousands of concurrent users. Pretty seamlessly, there's a user interface that a billion people on the planet use every month and are familiar with. And you can leverage that so your users have an experience they're familiar with. And one of the things that Google and other big web companies take seriously is in all device orientation. Making it work on tablets or laptops or Macs or PCs or Firefox or Safari just happens because that's part of how that company has been built. So that's kind of my opening. And what I'm going to do is walk through four use cases and basically just sort of say, you know, what is the solution about? Why was hybrid chosen? And then give you a peek at how it looks. So the first one is a pretty simple one. It's a project we did in association with a company that uses a bunch of Google technology. There are a newspaper conglomerate in Ontario, Canada. And one of the things they have is a big online real estate listing. And one of the things that they do with their online real estate listing is define neighborhood so people can search for listings in a particular neighborhood. And neighborhood mapping is complex and, you know, where are neighborhood boundaries? They choose to do it as groupings of census blocks. And they're constantly changing and refining them. So we're going to make this neighborhood bigger so we've got to add in this census block or we're going to add a new neighborhood and cut it out of these three other neighborhoods or whatever the case may be. So they need to do basically polygon aggregation out of Canadian census geometry. And one of the problems is Google Maps Engine does not support, which is their publication platform does not support polygon aggregation. So how do you add it? Perfect opportunity for a hybrid solution. And we built a capability that does that and uses post-gis to do that kind of analysis. And I'll sort of walk you through, I don't know if this cord is long enough. I'll try and talk about it. Just walk you through how the solution works. So essentially they build an aggregation file that says these are the polygons we want to be in every neighborhood. We put it on a web server. The core of what we built sits on a Linux machine with a Node.js component. It gets a new aggregation process. Okay, I got to do something. It goes to post-gis, initiates the aggregation routine. There's a new neighborhood layer available, pushes it back to this machine. And then this machine brokers a conversation with Google and replaces the old neighborhoods with the new neighborhoods. And seamlessly they're served to the public at Google scale and with the Google-based user interface that they've had for quite a long time. That's a hybrid system. It's got post-gis, it's got some Node, it's got some Google, and they all work together. So the second example is a little bit deeper. This project we did for the Western Governors Association called the Crucial Habitat Assessment Tool. And essentially it's the publication of a data set on which habitats in the west are most important from a variety of criteria. They ran a model and if it's darker blue it's more important and they're making these data available for people who are making land use choices or development choices or want to do cross-state analysis. There are 17 states in the west. So they put out an RFP to have their solution built. And one of the things that's interesting about the WGA is they don't have a GIS team. They don't have any GIS software. They don't, but they needed a GIS solution. And so that was, you know, so the landscape was wide open for the technology. It wasn't like, we're doing this, make it work on our thing. They were like, give us a good solution. And so we sort of had the freedom to propose what we thought was the best solution. And it ended up being a hybrid of boundless open geo, some open source stuff, Google, and Esri. And I'll kind of start from the bottom. The core of this data set is a giant grid of the west of about five and 10 kilometer cell hexagonal grids. And they're about 1.8 million grids. So it's a pretty big data set. And each state takes care of their own piece. And so the University of Kansas was part of the team and they managed putting all the puzzle pieces together to create one giant grid out of the component pieces. And that's done. And again, the Esri tools are used in that environment. Google was chosen for the JavaScript framework. Again, a good base map, imagery across the west, serve very fast, geo coding. You know, I can type in yellowstone. It takes me to Yellowstone National Park, those kinds of things. And it can play nice with other services. And then we had to serve the custom layer on top of that, this 1.8 million hexagons, color coded, and allow people to click on the hexagons and get the attributes or draw shapes and get all the hexagons into summary statistics. And so first we made a bunch of tiles for the west. So it remains fast, as fast as the Google base map, put them in Amazon. And then geo server and post just are used to manage the geometry and the query of the geometry. And this is what it looks like architecturally. And again, I can walk you through it. The Esri stuff is down here, our GIS, to put together the hexagons, Arc to Earth, helped us make the tiles along with your webcast. We push the tiles to Amazon, we push the hexagons, the post just. And then we have Web server code that manages the interface, calls the Google, you know, serves the JavaScript, etc. Serves the JavaScript, etc. Manages the user interface, and when people are doing various clicks that involve data, activates geo server or post just to get the attributes to highlight the polygons, etc. that they've selected. And it's been up for about a year and a half and running really, really flawlessly and with good performance, managed our launch spike on day one when ten governors mentioned it in their press releases. And then the last one that I want to talk about is a site called drivetexas.org. It's a public traveler information system in Texas. Google technology was chosen for scalability. Their sort of nightmare use case is a hurricane in the Gulf of Mexico, maybe Galveston, maybe even Houston need to be evacuated. And potentially you'd have a whole lot of users like hundreds of thousands or millions hitting it at the same time. And indeed this application was stress tested with 100,000 concurrent users, not just 100,000 connected 100,000 hitting the button at the same time. And no easy task. And again, the Google infrastructure performed very well. But the road conditions, the DOT has been maintaining road conditions data for a very long time in the Esri environment. And so the trick was how do you get the road conditions data into the Google infrastructure on a reliable, regular basis. And to me, this is one of my, the funner parts of the app, you get a splash screen when you come in and it tells you when the data was updated. And you'll see it was one minute before I happened to take the screen shot. It will not be ever more than 10 minutes. And that's basically this routine running silently in the background following this workflow. So the DOT has a mature environment as custom Esri tools, desktop tools for maintaining the road conditions. They have an on-premise database of those conditions and they publish those out with ArcGIS server. We built a component, runs on Linux, it's a Node.js component that accesses the Esri services to ID, any changes in the conditions from the last time it was updated. And then again pushes those changes up to Google Maps Engine so that the current data is available to be served at Google scale. And, you know, so again, there's no open source per se. It's a hybrid of basically Google and Esri. But the thing that we built, the Node.js tool, has been made available open source. Not the whole thing. There's a lot of very specific details that pertain just to that use case. But the guts of it for taking data in Node and pushing it to Google Maps Engine is openly available under the MIT license at NPM and through our Git repository. If this was clear, you'd see that the authors, CMWA, Calvin Metcalf who's sitting in the audience. And, you know, and in fact we use this exact same component in the MetroLand project that I referred to earlier. So, you know, you try and share and share alike. And we've averaged a lot of good work from other people in building the Node tool and we want to make that Node tool contribution that we added equally available. And then here's the last one, MapGeo migration. MapGeo is a hosted solution we offer to local governments. It's about 75 cities, towns, counties, mostly in the northeast, but as far as the way as the U.S. Virgin Islands. Use MapGeo to publish their data. And two PhosphorG North Americas ago, I did a whole talk on the architecture of MapGeo. It's a great tool. It's built entirely with GeoServer, GeoWebCache, and PostGIS. But one of the things we have to do is lots of tiling. We have to tile all the data for these 70 customers. And it's a complicated ballet and a bit of a hassle. So, as Google Maps Engine has evolved, we are moving the sort of tiling piece to Google Maps Engine. Basically, we don't have to tile. We move our data up there and Google serves it very fast in a tiled fashion. But Google Maps Engine doesn't do things like polygon buffers and intersect queries and things like that, you know, to do one of the capabilities is in a butters list. So even as we migrate parts of this away from an open solution, PostGIS will very much remain in the mix for, again, geometry serving, geospatial queries, et cetera. And, again, it's kind of the best of both worlds. We have easy tiling, fast tiled delivery, but we have the ability to do more GeoGeek queries, et cetera, by using PostGIS. So, you know, that wraps up the use cases. And really, you know, my take-homes are there are several models for hybrid. You know, it's not one size fits all. You can mix and match. And, you know, as Victoria Comayjin said, you know, picking the best solution, the best fitness for purpose to meet the customer's needs. And as you'll see, one of the things we've been doing, particularly with Google's platform, is it's a cost-effective way of filling platform or commercial gaps. If you don't have what you need in the commercial solution, you can add it yourself. And as Paul Ramsey observed, it can, doing these kinds of things can be the gateway. Get your feet wet with open source. See how it works. See how it works alongside your existing commercial solutions. And, again, you know, all of you are here. You get it. When you're doing these things in the open environment, don't be afraid to share, share what you're doing and make available your contributions, your additions to the projects you're working on. And I really appreciated what, you know, what both Sarah Novotnian and Mike Bostek said is, you know, Calvin, you know, contributes code, but there are other parts of our company that can contribute documentation and bug reports, et cetera, and staying involved with the projects that you access and leverage for your own business. And I think I have time for a few questions if anyone has any. Yes, sir. Michael, local governments were also sort of a hybrid shock when you use, you know, an enterprise, ESRI shop, so most of our data editing, like many local governments, is ESRI. But we also use Post and GeoServer or GeoWebCache, and now we're doing GME as well. But one of the things we've run into is getting data from SDE, you know, in some sort of ETL format, into GME. And using the GME APIs, we haven't found a way to do that yet. Are you using transactional updates to GME, or is it a true ETL loading the entire layer every 10 minutes? How does that work? I can refer to Calvin, but yeah, you have to work it out, and some things aren't fully mature, and doing it is real. Live from SCE, there's some way you've got to pull it out of ESRI and put it on GME. Yeah, there are some limits in GME that I hit repeatedly, and then I'd sort of like go complain to the person with Google, and then they come back and be like, oh yeah, yeah, yeah, you can't do that. So it ended up being that batch them as much as possible, ads before the leads, because you can't update something. Yes, which is the other fun thing. So it ended up being, I can talk to you more, what exactly black magic I had to do to get that to work, but hang in there. You can do it. Did you guys find that the solutions were becoming unnecessarily more complex because you had to do like an ETL, because you had to do like an ETL over just consuming services, or what, do you have experiences on that? It's a totally fair question, but it hinges on your definition of unnecessarily. This was necessary. They had an existing solution that could not handle the scale, was failing with a few hundred users in snowstorms in West Texas. They needed that piece, and so we had to just work through the problem. But geospatial people are pretty good at handling complexity. It's one of those things about ESRI. If you've been in the ESRI world for a long time, I started in the ESRI world in 1986 on a prime computer, you know, mini computer. You're pretty good at complexity, and you're dealing with ArcMap and ArcGIS server and ArcGIS online. They're all different components that you have to make work, and they do a reasonable job of making their components work together, but it's hardly simple, and so it's a judgment that you have to make. And my point is, don't do it this way just because I say so, but don't be afraid to look at alternative models and assess hybrid, how hybrid may or may not fit for your, for what, the problem you're trying to solve. It's just so frustrating for us to inevitably Google and solve this problem in a week and spend, you know, a month trying to work around the current issues, and then they're just going to release an update in their API. Sorry. It's the early days. I mean, there's no question Google's enterprise offerings are at version one, and there's a way to go, but they do work, and there's some other third party things. We might have well looked into using something like FME, which handshakes with GME. We had to build this project in six weeks before the start of hurricane season, and they weren't licensed FME users, etc., etc. So it's...it was quick. Two quick things. One, I think, if you're someone who's advocating to bring more things into your organization, you need to follow that best tool for the job or best of breed thing that you don't want to be the guy that brought in some really crappy open source thing because it was open source, and the managers see it and then say, it tainted for all projects. But to me, the open gateway drug in the Esri enterprise world is you need to replace your spatial database, use SDE, but this time use Postgres as the back end. And then even though you Ezrafy it, you can still get at geometries, you can still do all of those things on the back end, just through Postgres, and you can do some cool things like add triggers for doing overlays, spatial overlays on insert and update, and you can also then hit it with other back end processes that don't need to go through the Esri stuff that are fast. Exactly. I agree entirely. This is the game plan. This is Paul Ramsey speaking. It was great. In the talk, some of you may have been there, it was GIS for managers that how many of you are managers, and about 10% of the room raise their hand. He goes, well, why are the rest of you here? And they're like, we have to convince our managers. That's where we are. We're at the early days. I think we have time for one more. Yeah. I'm an aspiring co-founder, and I work at New Zealand on the Canterbury earthquake rebuild. And our problems is trying to merge data from lots of different organizations that are trying to work together. So it's essentially an SDI project. And what I'm kind of curious on the approach here, because what we face all the time is addressing standards and how that the various software vendors don't implement standards in the same way they say they support WFS, but the versions differ and their implementations differ. And so one of the standard, one of the tracks we take is that we don't really care what platform you use behind the scenes, but adhering to standards to make it as interoperable as possible. And I was just kind of wondering any of comments on that. I agree entirely with that. And Google publishes out of GME with a good WMS, a very fast standards compliant 1.3 WMS. Obviously, GeoServer has a very nice WMS. Esri has a decent WMS, although how up to date is changes with different releases. But I think that's a very smart thing. If you're facing the public and you want other people to consume it, make sure you put it out there in a consumable standard fashion. Very true. Thank you very much.
 Good morning.  My name is Michael Turner.  I'm one of the partners in applied geographic.  And I guess in the spirit of free and open source, it's open moderation here.  So I'm going to just dive in and talk for about 20 minutes on hybrid systems.  I think the place to start is with the definition of hybrid, at least my definition of hybrid.  And essentially, any solution that requires more than one piece of technology, one flavor  of technology, one stack of technology, and those technologies may be open or commercial  or both.  And I really like this quote from a blog that Esri did a few years back.  Those of you who have seen a recent blog posting, I did.  I quoted this as well.  But the real point is that the technology doesn't matter.  What really matters is what are you trying to do, and you're trying to pick the best  tools for getting it done.  And it doesn't all have to be one tool.  They can be intermixed.  I really appreciate that aesthetic.  Another way of thinking about it is it's best of breed.  Don't be afraid to mix and match things because some things are very good for one thing or  the other thing.  You can have the best geocoder and the best map publishing environment and the best desktop  environment if that's part of your solution.  Just as with dogs and where best of breed comes from, some dogs are good in the water,  some dogs are good hunting, some dogs are good herding.  You get the one that you need.  But it's a big marketplace.  Our geospatial environment is a big marketplace.  And it's hard or can be difficult to make choices when there are so many choices, you  know, and so many pieces of technology, whether it's desktop or JavaScript frameworks or geospatial  servers or even database servers.  And then what's best commercial, what's best open source.  So it's not a simple task to sort of implement best of breed.  You have to be a good consumer or a smart shopper, if you will.  I had this slide this morning after seeing Paul Ramsey's talk on post-just for managers.  I think it's a real important set of observations that it's not an all or nothing that sometimes  starting with a small constrained open source component can get the ball rolling.  And what he was saying in the context of post-just but substitute any geospatial open source is  maybe you have two things for a little while.  Your staff get used to it.  They build confidence and then bit by bit you migrate or build a dual capacity.  And you know, it's important to also, you know, respect where commercial software came  from.  Just because open source is readily available, readily downloadable, the savings aren't immediately  there.  Most commercial software is paid for, was paid for a long time ago.  And there are real business costs to switching.  And economic term, a switching cost can be non-trivial if you have a big existing mature  operation.  And again, you know, a few more details on the commercial side.  You know, we've been around 23 plus years and, you know, I grew up in an environment  in the, in the night, you know, professionally grew up in an environment in the 1980s where  there weren't geospatial open source alternatives.  So a lot of people have commercial stuff because they started a long time ago.  And it's not just the costs and money.  Organizations have a lot of expertise and staff knowledge about how the commercial systems  work.  And in my opinion, there are, you know, some of the commercial systems remain the best  of greed.  If you want to do real high-end cartography, if you want to do sort of esoteric geospatial  analysis, et cetera, you might judge that commercial systems are very, very competitive.  And obviously, one of the things that's starting to emerge in the open source system, ecosystem,  is a support network.  Places to go for help, places where you can buy support and things of that nature.  And again, in commercial, that's always been there.  That's one of the things you very much are buying.  So that's sort of a view of some of the things that Esri really does well and the core of  their business.  And then I'm going to talk about some Esri stuff and some Google stuff.  And on Google, what people are looking for is the scalability of their cloud, the ability  to have applications that can support hundreds of thousands of concurrent users.  Pretty seamlessly, there's a user interface that a billion people on the planet use every  month and are familiar with.  And you can leverage that so your users have an experience they're familiar with.  And one of the things that Google and other big web companies take seriously is in all  device orientation.  Making it work on tablets or laptops or Macs or PCs or Firefox or Safari just happens because  that's part of how that company has been built.  So that's kind of my opening.  And what I'm going to do is walk through four use cases and basically just sort of say,  you know, what is the solution about?  Why was hybrid chosen?  And then give you a peek at how it looks.  So the first one is a pretty simple one.  It's a project we did in association with a company that uses a bunch of Google technology.  There are a newspaper conglomerate in Ontario, Canada.  And one of the things they have is a big online real estate listing.  And one of the things that they do with their online real estate listing is define neighborhood  so people can search for listings in a particular neighborhood.  And neighborhood mapping is complex and, you know, where are neighborhood boundaries?  They choose to do it as groupings of census blocks.  And they're constantly changing and refining them.  So we're going to make this neighborhood bigger so we've got to add in this census block or  we're going to add a new neighborhood and cut it out of these three other neighborhoods  or whatever the case may be.  So they need to do basically polygon aggregation out of Canadian census geometry.  And one of the problems is Google Maps Engine does not support, which is their publication  platform does not support polygon aggregation.  So how do you add it?  Perfect opportunity for a hybrid solution.  And we built a capability that does that and uses post-gis to do that kind of analysis.  And I'll sort of walk you through, I don't know if this cord is long enough.  I'll try and talk about it.  Just walk you through how the solution works.  So essentially they build an aggregation file that says these are the polygons we want  to be in every neighborhood.  We put it on a web server.  The core of what we built sits on a Linux machine with a Node.js component.  It gets a new aggregation process.  Okay, I got to do something.  It goes to post-gis, initiates the aggregation routine.  There's a new neighborhood layer available, pushes it back to this machine.  And then this machine brokers a conversation with Google and replaces the old neighborhoods  with the new neighborhoods.  And seamlessly they're served to the public at Google scale and with the Google-based user  interface that they've had for quite a long time.  That's a hybrid system.  It's got post-gis, it's got some Node, it's got some Google, and they all work together.  So the second example is a little bit deeper.  This project we did for the Western Governors Association called the Crucial Habitat Assessment  Tool.  And essentially it's the publication of a data set on which habitats in the west are  most important from a variety of criteria.  They ran a model and if it's darker blue it's more important and they're making these data  available for people who are making land use choices or development choices or want to  do cross-state analysis.  There are 17 states in the west.  So they put out an RFP to have their solution built.  And one of the things that's interesting about the WGA is they don't have a GIS team.  They don't have any GIS software.  They don't, but they needed a GIS solution.  And so that was, you know, so the landscape was wide open for the technology.  It wasn't like, we're doing this, make it work on our thing.  They were like, give us a good solution.  And so we sort of had the freedom to propose what we thought was the best solution.  And it ended up being a hybrid of boundless open geo, some open source stuff, Google,  and Esri.  And I'll kind of start from the bottom.  The core of this data set is a giant grid of the west of about five and 10 kilometer  cell hexagonal grids.  And they're about 1.8 million grids.  So it's a pretty big data set.  And each state takes care of their own piece.  And so the University of Kansas was part of the team and they managed putting all the  puzzle pieces together to create one giant grid out of the component pieces.  And that's done.  And again, the Esri tools are used in that environment.  Google was chosen for the JavaScript framework.  Again, a good base map, imagery across the west, serve very fast, geo coding.  You know, I can type in yellowstone.  It takes me to Yellowstone National Park, those kinds of things.  And it can play nice with other services.  And then we had to serve the custom layer on top of that, this 1.8 million hexagons,  color coded, and allow people to click on the hexagons and get the attributes or draw  shapes and get all the hexagons into summary statistics.  And so first we made a bunch of tiles for the west.  So it remains fast, as fast as the Google base map, put them in Amazon.  And then geo server and post just are used to manage the geometry and the query of the  geometry.  And this is what it looks like architecturally.  And again, I can walk you through it.  The Esri stuff is down here, our GIS, to put together the hexagons, Arc to Earth,  helped us make the tiles along with your webcast.  We push the tiles to Amazon, we push the hexagons, the post just.  And then we have Web server code that manages the interface, calls the Google, you know,  serves the JavaScript, etc.  Serves the JavaScript, etc.  Manages the user interface, and when people are doing various clicks that involve data,  activates geo server or post just to get the attributes to highlight the polygons, etc.  that they've selected.  And it's been up for about a year and a half and running really, really flawlessly and with  good performance, managed our launch spike on day one when ten governors mentioned it  in their press releases.  And then the last one that I want to talk about is a site called drivetexas.org.  It's a public traveler information system in Texas.  Google technology was chosen for scalability.  Their sort of nightmare use case is a hurricane in the Gulf of Mexico, maybe Galveston,  maybe even Houston need to be evacuated.  And potentially you'd have a whole lot of users like hundreds of thousands or millions  hitting it at the same time.  And indeed this application was stress tested with 100,000 concurrent users,  not just 100,000 connected 100,000 hitting the button at the same time.  And no easy task.  And again, the Google infrastructure performed very well.  But the road conditions, the DOT has been maintaining road conditions data for a very long time in the  Esri environment.  And so the trick was how do you get the road conditions data into the Google infrastructure  on a reliable, regular basis.  And to me, this is one of my, the funner parts of the app, you get a splash screen when you come in  and it tells you when the data was updated.  And you'll see it was one minute before I happened to take the screen shot.  It will not be ever more than 10 minutes.  And that's basically this routine running silently in the background following this workflow.  So the DOT has a mature environment as custom Esri tools, desktop tools for maintaining the road conditions.  They have an on-premise database of those conditions and they publish those out with ArcGIS server.  We built a component, runs on Linux, it's a Node.js component that accesses the Esri services to ID,  any changes in the conditions from the last time it was updated.  And then again pushes those changes up to Google Maps Engine so that the current data is available to be served at Google scale.  And, you know, so again, there's no open source per se.  It's a hybrid of basically Google and Esri.  But the thing that we built, the Node.js tool, has been made available open source.  Not the whole thing.  There's a lot of very specific details that pertain just to that use case.  But the guts of it for taking data in Node and pushing it to Google Maps Engine is openly available under the MIT license  at NPM and through our Git repository.  If this was clear, you'd see that the authors, CMWA, Calvin Metcalf who's sitting in the audience.  And, you know, and in fact we use this exact same component in the MetroLand project that I referred to earlier.  So, you know, you try and share and share alike.  And we've averaged a lot of good work from other people in building the Node tool and we want to make that Node tool contribution that we added equally available.  And then here's the last one, MapGeo migration.  MapGeo is a hosted solution we offer to local governments.  It's about 75 cities, towns, counties, mostly in the northeast, but as far as the way as the U.S. Virgin Islands.  Use MapGeo to publish their data.  And two PhosphorG North Americas ago, I did a whole talk on the architecture of MapGeo.  It's a great tool.  It's built entirely with GeoServer, GeoWebCache, and PostGIS.  But one of the things we have to do is lots of tiling.  We have to tile all the data for these 70 customers.  And it's a complicated ballet and a bit of a hassle.  So, as Google Maps Engine has evolved, we are moving the sort of tiling piece to Google Maps Engine.  Basically, we don't have to tile.  We move our data up there and Google serves it very fast in a tiled fashion.  But Google Maps Engine doesn't do things like polygon buffers and intersect queries and things like that,  you know, to do one of the capabilities is in a butters list.  So even as we migrate parts of this away from an open solution, PostGIS will very much remain in the mix  for, again, geometry serving, geospatial queries, et cetera.  And, again, it's kind of the best of both worlds.  We have easy tiling, fast tiled delivery, but we have the ability to do more GeoGeek queries, et cetera,  by using PostGIS.  So, you know, that wraps up the use cases.  And really, you know, my take-homes are there are several models for hybrid.  You know, it's not one size fits all.  You can mix and match. And, you know, as Victoria Comayjin said, you know,  picking the best solution, the best fitness for purpose to meet the customer's needs.  And as you'll see, one of the things we've been doing, particularly with Google's platform,  is it's a cost-effective way of filling platform or commercial gaps.  If you don't have what you need in the commercial solution, you can add it yourself.  And as Paul Ramsey observed, it can, doing these kinds of things can be the gateway.  Get your feet wet with open source. See how it works.  See how it works alongside your existing commercial solutions.  And, again, you know, all of you are here. You get it.  When you're doing these things in the open environment, don't be afraid to share,  share what you're doing and make available your contributions, your additions to the projects you're working on.  And I really appreciated what, you know, what both Sarah Novotnian and Mike Bostek said is, you know,  Calvin, you know, contributes code, but there are other parts of our company that can contribute documentation  and bug reports, et cetera, and staying involved with the projects that you access  and leverage for your own business.  And I think I have time for a few questions if anyone has any.  Yes, sir.  Michael, local governments were also sort of a hybrid shock when you use, you know,  an enterprise, ESRI shop, so most of our data editing, like many local governments, is ESRI.  But we also use Post and GeoServer or GeoWebCache, and now we're doing GME as well.  But one of the things we've run into is getting data from SDE, you know, in some sort of ETL format, into GME.  And using the GME APIs, we haven't found a way to do that yet.  Are you using transactional updates to GME, or is it a true ETL loading the entire layer every 10 minutes?  How does that work?  I can refer to Calvin, but yeah, you have to work it out, and some things aren't fully mature, and doing it is real.  Live from SCE, there's some way you've got to pull it out of ESRI and put it on GME.  Yeah, there are some limits in GME that I hit repeatedly, and then I'd sort of like go complain to the person with Google,  and then they come back and be like, oh yeah, yeah, yeah, you can't do that.  So it ended up being that batch them as much as possible, ads before the leads, because you can't update something.  Yes, which is the other fun thing.  So it ended up being, I can talk to you more, what exactly black magic I had to do to get that to work, but hang in there.  You can do it.  Did you guys find that the solutions were becoming unnecessarily more complex because you had to do like an ETL,  because you had to do like an ETL over just consuming services, or what, do you have experiences on that?  It's a totally fair question, but it hinges on your definition of unnecessarily.  This was necessary.  They had an existing solution that could not handle the scale, was failing with a few hundred users in snowstorms in West Texas.  They needed that piece, and so we had to just work through the problem.  But geospatial people are pretty good at handling complexity.  It's one of those things about ESRI.  If you've been in the ESRI world for a long time, I started in the ESRI world in 1986 on a prime computer, you know, mini computer.  You're pretty good at complexity, and you're dealing with ArcMap and ArcGIS server and ArcGIS online.  They're all different components that you have to make work, and they do a reasonable job of making their components work together,  but it's hardly simple, and so it's a judgment that you have to make.  And my point is, don't do it this way just because I say so, but don't be afraid to look at alternative models and assess hybrid,  how hybrid may or may not fit for your, for what, the problem you're trying to solve.  It's just so frustrating for us to inevitably Google and solve this problem in a week and spend, you know, a month trying to work around the current issues,  and then they're just going to release an update in their API.  Sorry.  It's the early days. I mean, there's no question Google's enterprise offerings are at version one, and there's a way to go,  but they do work, and there's some other third party things.  We might have well looked into using something like FME, which handshakes with GME.  We had to build this project in six weeks before the start of hurricane season, and they weren't licensed FME users, etc., etc.  So it's...it was quick.  Two quick things. One, I think, if you're someone who's advocating to bring more things into your organization,  you need to follow that best tool for the job or best of breed thing that you don't want to be the guy that brought in some really crappy open source thing  because it was open source, and the managers see it and then say, it tainted for all projects.  But to me, the open gateway drug in the Esri enterprise world is you need to replace your spatial database, use SDE,  but this time use Postgres as the back end.  And then even though you Ezrafy it, you can still get at geometries, you can still do all of those things on the back end,  just through Postgres, and you can do some cool things like add triggers for doing overlays, spatial overlays on insert and update,  and you can also then hit it with other back end processes that don't need to go through the Esri stuff that are fast.  Exactly.  I agree entirely. This is the game plan. This is Paul Ramsey speaking.  It was great. In the talk, some of you may have been there, it was GIS for managers that how many of you are managers,  and about 10% of the room raise their hand. He goes, well, why are the rest of you here?  And they're like, we have to convince our managers. That's where we are. We're at the early days.  I think we have time for one more.  Yeah. I'm an aspiring co-founder, and I work at New Zealand on the Canterbury earthquake rebuild.  And our problems is trying to merge data from lots of different organizations that are trying to work together.  So it's essentially an SDI project.  And what I'm kind of curious on the approach here, because what we face all the time is addressing standards  and how that the various software vendors don't implement standards in the same way they say they support WFS,  but the versions differ and their implementations differ.  And so one of the standard, one of the tracks we take is that we don't really care what platform you use behind the scenes,  but adhering to standards to make it as interoperable as possible.  And I was just kind of wondering any of comments on that.  I agree entirely with that. And Google publishes out of GME with a good WMS, a very fast standards compliant 1.3 WMS.  Obviously, GeoServer has a very nice WMS.  Esri has a decent WMS, although how up to date is changes with different releases.  But I think that's a very smart thing. If you're facing the public and you want other people to consume it,  make sure you put it out there in a consumable standard fashion.  Very true. Thank you very much.
01_Michael_Terner_720p_0_107.png 01_Michael_Terner_720p_108_479.png 01_Michael_Terner_720p_480_1091.png 01_Michael_Terner_720p_1092_1859.png 01_Michael_Terner_720p_1860_2795.png 01_Michael_Terner_720p_2796_3791.png 01_Michael_Terner_720p_3792_5555.png 01_Michael_Terner_720p_5556_8819.png 01_Michael_Terner_720p_8820_9239.png 01_Michael_Terner_720p_9240_10523.png 01_Michael_Terner_720p_10524_11171.png 01_Michael_Terner_720p_11172_12983.png 01_Michael_Terner_720p_12984_13979.png 01_Michael_Terner_720p_13980_14831.png 01_Michael_Terner_720p_14832_17387.png 01_Michael_Terner_720p_17388_19175.png 01_Michael_Terner_720p_19176_20891.png 01_Michael_Terner_720p_20892_21455.png 01_Michael_Terner_720p_21456_22631.png 01_Michael_Terner_720p_22632_24395.png 01_Michael_Terner_720p_24396_27083.png 01_Michael_Terner_720p_27084_29387.png 01_Michael_Terner_720p_29388_38111.png 01_Michael_Terner_720p_38112_38159.png 01_Michael_Terner_720p_38160_38207.png 01_Michael_Terner_720p_38208_38255.png 01_Michael_Terner_720p_38256_38303.png 01_Michael_Terner_720p_38304_38351.png 01_Michael_Terner_720p_38352_39071.png 01_Michael_Terner_720p_39072_42215.png 01_Michael_Terner_720p_42216_42217.png