OGC GeoPackage in practice: Implementing a new OGC specification with open-source tools
CC Attribution - NonCommercial - ShareAlike 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Frantz, Nathan
GeoPackage is a new encoding standard created by the Open Geospatial Consortium as a modern alternative to formats like SDTS and Shapefile. Using SQLite, the single-file relational database can hold raster imagery, vector features and metadata. GeoPackage is an ideal data container for mobile devices such as smartphones, IoT devices, wearables, and even automobiles. We have created a few open-source tools to manipulate this exciting technology in a way that is useful to the geospatial community. Our goal with the GeoPackage specification implementations is simple: Create GeoPackages quickly and reliably while maintaining standard conformance. The single biggest issue we have faced is the speed in which large amounts of imagery can be disseminated to the end user. Data standards reliability was also a concern because we found many vendors interpreted the specification differently or to suite their own needs. Finally, the main problem GeoPackage was created was to solve was interoperability. We set out to create an implementation that would guide other parties towards making a data product that would function as well on one platform as it would on a completely different platform. Our initial implementation of the GeoPackage specification was created using Python 2.7.x. The software design was intended for command line use only in a script-friendly environment where tiling speed was paramount. The Gdal2tiles.py script was improved upon by harnessing the Python multiprocessing library so that multiple tile jobs could run simultaneously. The other piece of the workflow, creating GeoPackages, would be a separate development effort from scratch called tiles2gpkg_parallel.py. In tiles2gpkg_parallel.py, we implemented multiprocessing by writing to separate SQLite databases in parallel and then merging the tiled data sets into one compact database. This implementation worked well and increased the performance of producing these data sets; however, the command line design means that all but the most technically adept users would struggle to use the tools. With the initial Python implementation getting early-adopters a preview of GeoPackage in the short term, our team set out to make a production-quality GeoPackage API that could satisfy all user needs. Named Software to Aggregate Geospatial Data or SWAGD, we created a robust library for tiling raster data, packaging raster data stores into GeoPackages, and viewing either the raw tiles OR the finished GeoPackage products within a map viewer. Additionally, a Geopackage verification tool was created to foster community adoption. For more information, see our Github site here: Many open-source tools are being leveraged on the SWAGD project, including many common build and continuous integration tools including Github, TravisCI, WaffleIO, and Coverity. Using proven software development mechanisms like unit testing and code reviews we now have a consistent, reproducible, and inclusive GeoPackage implementation. We have an aggressive list of future capability that we would like to develop including ad-hoc routing on a mobile device, vector tile data sets, and even 3D support.
 All right, it's about 11.25 now, so I'll get started. Hi there, everyone. I'm Steve Lander. I'm a software engineer at RGI, and I'm going to talk to you about implementing OGC Geo package. OGC Geo package being a new standard that OGC just came out with recently, and this is for a containing feature, raster data, meta data, schemas, all kinds of stuff into a SQLite database. So a little bit more about me. I've worked out of Fairfax, Virginia in the United States. I'm experienced in caching and storing large raster images, so the work I've been doing for, I would say, the past three years has been involved with using GDAL quite often to take these large raster images, these geospatially referenced images, and then cutting them into many pieces. And there are a lot of benefits for doing that, main benefits being that it greatly increases the time it takes to actually render that information. So, and the work we've been doing for the past three years has been implementing that encoding standard. We initially started off with Python, that progressed into Java, and then from Java, we also wanted to put it in Android. There's been some others as well, JavaScript, but it hasn't really gone anywhere. So our actual GitHub projects, if you want to know more, software to aggregate geospatial data, otherwise known as SwagD, and GeoPackagePython, which was the first naive implementation we started off with. So for SwagD, we were starting off with Java 1.8, which was kind of aggressive. If I was to do it again, I'd probably start off maybe with Java 1.7 or Java 1.6, just because not a lot of people are comfortable with going to the latest version of Java. Typically, you'll find a lot of people with 1.6 or 1.7 out in the wild, and they're kind of reluctant to go any further into new stuff. But, you know, there's a lot of benefits for going to Java 1.8 that we saw that we just decided to dive in and see how it went. So, if you want to see some other presentations we have going on yesterday, and I'm sure it's going to be on the website, Nathan France had talked about GeoPackage and how that's changed the way the governments think about standards, and also tomorrow I'll be presenting some of the open source build tools that I have incorporated into my projects in order to help me and help my team, you know, reduce our risk, reduce our errors. So, what is the GeoPackage Encoding Standard? And we're also going to be talking about the actual implementation we've made and our approach to implementing the encoding standard and what we learned building the reference implementation. And the lessons learned from this experience, they are numerous. So, GeoPackage Encoding Standard is a set of conventions for storing the following, and this is all available at the OTC website. So, vector features I mentioned, a tile matrix set, which is that GeoReference image cut into many pieces, and we do that at many scales. So, it's like taking a basically a pyramid data set and throwing it into a SQLite database. There's also a lot of schema attached to that. Describing your tile set, describing what levels it exists at, how many there are, and a lot of other helpful things that you don't really get with MB tiles. MB tiles just kind of dumps a lot of tiles into a single SQLite database. MB tiles likes to throw some rigor around that so that you have more information, more metadata. So, yeah, speaking of metadata, you know, you can describe your, the type of data you have more. Maybe you want to talk about a classification or, you know, the sky's the limit. And then, extension. So, the extension says, I think is one of the more powerful features of it. Extension will allow you to do things that the current standard, the way it's written now, will not let you. So, things like perhaps you want to add a network data set into your GeoPact. Perhaps you want to add time stamping to your images, to your tiles. Because tiles can update, tiles can change, and you want to figure out which ones are the freshest or the most stale, for instance. So, when we started, and this is something that Vladimir actually had too, if you saw Vladimir Agafonkin's presentation. So, how do these, why do we have another standard for doing this? I think it's a little bit better than the situation here is saying because I saw that MB tiles was pretty good. MB tiles was a really good way to take this idea of a cache map and throw it onto a mobile device or any device as a single file. And I thought there was a lot of, a lot of good value there. But also, you don't have to take the entirety of GeoPact. You don't have to put rectors in there if you don't want to. You don't have to put rasters in there if you don't want to. You can make a blank one if you really desire to. But I think that there's nothing wrong with more standards, but it's going to really depend on your use case. So, what kind of existing technology is already out there that perhaps could compete with GeoPact or does something similar to what GeoPact does? The main one you'll see on OGC site is the shapefile. So, shapefile, you'll have your vector information, you'll have your metadata, but the shapefile is not going to do your raster tile data sets. KML would kind of fit that need a little bit better than shapefile. You would actually be able to take your tile map data sets in KML and view them, and also vector as well. But you don't get some of the nicer parts of the scheme of the metadata and the extensions with KML. GeoJSON, very fast, very good for vector. And then NPTiles, you could probably throw vector information into an NPTiles, but the problem there being, as a spec, NPTiles only supports Web Mercator. Whereas GeoPact, you can support any EPSG code or anything specific, even your own custom WKB or WKT if you so want it. So, there are more implementers of GeoPact and actually growing than just ours. GDAL had that quite a while ago for just features in 111.0. We actually use 111 in our software now. And raster tiles have just emerged in 2.0. We haven't looked at that. I haven't looked at that personally, but it's pretty exciting. GeoServer, I talked to some of the GeoServer people that are here. There's a community supported plugin. That's not everybody's radar, but nothing actually official yet, as far as I know. Please correct me if I'm wrong. So, Spatial Light, they just added the version. I checked that probably a month ago, and that was just pretty recent. Esri has had feature support, I would say, and their supported GeoPact is actually still ongoing. Raster support is coming, however, you know, the way they deal with tile image, tiles and tile in a cache is actually fundamentally different from your TMS or your WMTS. They actually have a binary format, tile cache type situation that doesn't make it a really quick fit. And then Digital Globe, they have the ability currently to push out the imagery that you see to a GeoPackage. You can actually export that. And the asterisk is there because that was code that we actually made. That's the Python naive implementation. So, if you have a Digital Globe account, you can go there and instead of getting a file folder structure full of tiles, you can actually just get a GeoPackage, have a SQLite database. And if your viewer supports it, you can see that. So, our approach. So, what I really wanted to do was create an API for other developers and for other implementers that they could just pick and choose what they wanted. So, I knew that the intended user that we were raking the software for wasn't going to need everything in a single project, right? You would have certain individuals that maybe would want to deploy this on a server. They would want to perhaps have users come in and they would have a tile back end on their server that they would be able to draw from, take those tiles or take that imagery, tile it there even and then push it out to a user that they could have a downloadable file from. Or maybe they want to view it on a mobile and tiling just doesn't make sense. And I wanted to build a UI around that API as just a demonstration so that you could see the types of things it would allow you to do if you use it on a desktop, if you use it on a server or if you use it on a mobile device or whatever. And I mentioned we started off with Java 1.8 and which is really great. Everybody on the team loves it but there are some problems. There are some features that especially for Android and we'll talk about that later just aren't going to work in and we had to backport some things. But that's how it goes. So the core API, we have a common library that shows your typical geospatial things. You have a data store library which is going to have the classes that allow you to manipulate a store of tiles. A store of tiles could be a file folder structure of TMS organized images or it could be a GIF package or it could be an MB tiles. Whatever you have that's storing tile information, cache map information, that's that library is going to allow you to interact with those. A GDALT2Tiles project as well exists in there and the name taken from the Python script by the same name and a lot of the code in that project closely follows what they're doing and that's basically just taking any GDALT supported image and tiling it for you and giving you output. The GeoPackage class is typically what people are going to want. That's going to allow you to verify a GeoPackage that's going to allow you to open one, to create one, to put imagery into one and then our network extension is actually an extension that we talked about where you're able to add a network dataset. You have to add for instance nodes and edges to a GeoPackage and interact and query that information on your actual device like maybe a mobile application or your laptop using things like ASTAR or Dykstra routing. We just had a keynote about open software. Interoperability is definitely something that I have to worry about constantly. I wanted to have that thought about before coming in and making everything. We had a verification component there that when you open the GeoPackage, it's going to check literally everything. All 200 pages of that GeoPackage encoding document, it's going to go through and see if you have been adhering to that. And then that was so popular that we actually had to make that a separate application so that other people in the wild, other implementers, other companies could take that and say, hey, I want to make my GeoPackage interoperable. Do you have anything that could help? And we're like, sure, this is this verification app. It will help you out and it will actually give you pointers to the actual specification headline so that you can say, okay, I'm not following the spec here and what do I do? What does this even mean? Another way we show for interoperability support was just using the JVM. I probably could have gone Python, I could have gone Haskell, I could have gone crazy, but the reality is the JVM has allowed, my development team to use Windows Linux, I personally use Linux to do my development on, has allowed us to push to Android. So that was very, very important that we have something that was easily portable. And the outputs that it supports are if you've ever made a leaflet-based map or if you've ever made a leaflet map, those are the big three that you'll have support for baked in. Three or five, seven more Mercator, global geodetic, eco-retangular, and then global Mercator, which you typically don't see a whole lot of people making global Mercator, but they give it as web Mercator only, not as egregious in the, in the sort of the distortion that it introduces. So we made a reference to UI that I mentioned. It's swing-based, which really makes me sad because I don't like swing at all. I really wish it was kind of an NBC, MVVM situation. But, you know, we're getting there. Java FX just came out for 1-8, so that's something that's very exciting for the team. JMapViewer is a, is a jar that I, that I happened to find through some research, and JMapViewer actually supported tile map sets out of the box, so I didn't have to read write anything. The problem is it was the only one that was out there. So there are some things that it does poorly that I wish it didn't do. And, and if we ever moved to a two Java FX there in that last headline, that JMapViewer would be going away because it's, it's not very, not very flexible, 3-8, 5-7 only. You know, some, some odd behavior. This is our reference UI here that I made. Simplicity. It was definitely the goal, three workflows, tiling data, packaging that data and then viewing it. What we're seeing here is the viewing. And then this is an Alaska data set pulled from, from a geo package. And you're actually viewing the tile map imagery there. Very, very basic, very plain. But when you have something like that available on, on like a mobile device that you don't have to worry about connectivity, it could be, you know, quite powerful. These are the, some of the inputs we take when we are doing tiling. For instance, you know, full, full reference system support incoming and then outgoing. We, we limit that. You'll see the drop down box there. You know, we, we limit that because of what we know a, a user, what we, what we assume a user will be able to, to want what they will want. You know, 3-8, 5-7, typically in 90% of the cases. And then, you know, 3-8, 5, or 4-3, 2-6, kind of, you know, tacit support right now that will be coming of, you know, actual rigorous support for that will be coming a little bit later. I'm making, you can also tile directly to a GA package. If you just want to skip the whole creating, creating images, you can do that and just tile directly to a GA package. Or if you want to have the flexibility of having all those loose tiles somewhere on your system, you can, you can do that as well. If you want to change what you're making, if you don't want to make a huge PNG collection, you want to just make a JPEG collection with some, with some compression, you can do that as well. And that's a metadata here at the bottom and where you want to save it. So, then when we actually package, you know, pretty similar, you can actually change the imagery that you have as well when you're packaging. You can take a PNG image because you want to save that transparency information and then convert it to a JPEG because maybe you don't, maybe space is more important, right? If you're targeting a mobile device, the difference between a GA package that is, that is PNG and JPEG can be, you know, many hundreds of megabytes. I mentioned the verifier, verifier is Java FX and a lot better, I think, of a UI than, than the swing application you just saw, unfortunately. But, you know, redoing an application, redoing the UI for the application that we have that you just saw is, is a large task. So, it tests for the following parts of the GA package encoding standard, all the, all the big ones that you have, the one that's noticeably absent there is features and features is just because we haven't gotten there yet. So, the idea of what it looks like, Alaska, Alaska 2.geopackage, which you just saw, was good. You can, you know, collapse that. You can see the other, other GA package and what it's past. You can also see, and I apologize, the radiant blue is the only GA package that I had on my system that would show a warning. But I think this is a good illustration in that you have a warning here, not necessarily a breakage. So, for instance, on this line here, it says, this is pulled directly from the spec. A GA package shall contain, you know, 047503130 GP10 in that scale. That really doesn't mean a whole lot to a layman, right? But, but all that's saying is, if you have this one byte header set in the binary file, it should be there, but it's not, you can still open it. It's not fatal. So, what did we learn building this reference, reference implementation? What have we learned about coding? What have we learned about ourselves? So, for Java, I realized really early on that, man, there are just very few choices for viewing tile data sets. I mentioned JMapViewer. That was really it. And finding that was on some, some Arcane website that I found that had a download link and I wasn't sure if it was malware or not, but it's not malware. So, I wish, I really wish we knew more about Java FX. I actually found out about Java FX from the last Phosphor G in San Francisco in Berlin and once I found out, I was like, wow, that's amazing. I wish, I wish really earlier heard about that sooner. A lot of many tiers could have been averted. The Python implementation that we have for tiling is still far in a way, the fastest way to make tiles. We researched in Java, you know, multi-threading, multi-processing. How can we, how can we speed this up? The Python implementation takes, uses the Python multi-processing library and it is so fast that it will kill your system. You'll use every single core and it won't be usable, but man, you'll have tiles really quickly. We can't get that with Java right now, unfortunately. That's because of the JVM, the limitations you have with the JVM. And you'll find this a lot when you, this last point, you'll find this a lot when you're dealing with, with database operations. Optimizing the way you interact with the database is typically giving you your biggest gains in performance. So reduce the amount of SQL calls you make and then reduce the amount of times you're going to your, your SQLite database or your database in particular because that's, that's going to take the longest time, right? Worrying about pieces of code in, in certain areas where you're maybe using a list versus a hash map, that's not going to get you the type of improvements you'll get versus going over a database. What have you learned about Android? Compatibility for Android for just a while. See, I was on the impression like, oh, you have Java, just throw it right on the Android, on the Android device and it'll just work. But that's not the case. It's still stuck in 1.6 officially. 1.7 support is, is there. I believe you can make a 1.7 Java project for Android and just bring it over and it, and it should work. But we didn't want to take that chance. So the fact that we had Java 1.8 and we had to go back for a couple, just a couple libraries to do the, the compatibility for Android, that really stunk. So, so the desktop development for, for Android is really ill-suited. The problem is when you're developing on a desktop for interacting with a SQLite database, those libraries and those, that functionality that you'll have available to you is going to differ than it is on an Android device. Android device having their own SQLite implementation that's kind of baked into Android, whereas if you're on the desktop, you have to, you have to bring that in. That's not something that you just get with Java. And then, well, big one, you know, a buffered image and a desktop Java versus a bitmap image on an, on an Android Java implementation. So you kind of have to have an adapter there to, to help you bridge that. And, and they're not necessarily very compatible. And then, the different testing frameworks that, that exist on, on, you know, JUnit versus an, an Android one on Android. And then, there was frustratingly no, no globally available, most current Android jar available like Maven. So we just had to, had to throw that in a source and reference it. So regarding the spec, you know, I, I thought I'd been working on this for three years. There's, there are still questions about the spec, still, still different interpretations on a single word. And that's kind of frustrating. Even though they tried to be as verbose and as precise as possible, you still have differing opinions and, and the ways that you can, you can do your, you can do your tileset. So compromises are the key. And I already mentioned, Lingokin differ. So in general, think about who you want to use your code and, and how, and then constrain that list. You saw that the UI had, it was actually gave you very, very few options. So, all right. Thank you for listening. Thank you, Steve. Do you have any questions for, for Steve? One of the problems with extensions is as soon as you start using extensions, you reduce who you can share this data with because extensions are not universally done. So. Sure. So the good part about extensions is if you're, if you're making a Geo package implementation, you will just ask the Geo package, what, what extensions do you have? And if it has an extension that you don't have, you still, you can still open it. You can still get your, your functionality that you want. You don't, you're not losing anything. You just, you know, if you get something with an extension that you don't support, you can still look at the tiles. You can still look at the stuff that you can still get all the things you get with a vanilla Geo package. True. If you, if you have an extension, you, that's going to be implementation specific. And that's, that's kind of a, that's kind of a shame. But hopefully you're registering your extension with OGC or you're registering it with the body so that you can, you can have people and you know, tie it back to the keynote. You know, you got to be as open as possible. And, and that'll help. So you have to mitigate, unfortunately. Sure. Sir? What was your use case for, for using Geo? Well, you are, what, what, you're only using in your application like we did on the online, just for the, on the third? The use case was we, we wanted fast base maps and fast data on a mobile device. So there, there were, you know, you could have MB tiles. MB tiles was great. MB tiles could work. However, more information we felt was better. The more rigor you throw around it. So that's why we went with Geo package. And third. Hmm? And third. Yeah. Yeah. Sir? Do you think that's the best use case for Geo package or is it better as kind of a universal standard for data interchange rather than a mobile device? It's a, that's a good question. I think that Geo package should exist outside of, outside of the mobile realm. I think the mobile realm is just one in which that it works particularly well. There are plenty that don't work very well. So for instance, storing your data, storing your tile map data on the server in Geo package, I would not recommend that because you will have problems with SQLite at a certain point. Eventually you'll get to a size, I mean, in the size can be terabytes of for SQLite, right? But eventually you get to a point where inserts no longer make sense. Working with that Geo package no longer makes sense. You might as well just have a big binary blob of tiles or file folder structure or something similar. Any others? Other question? I think we have one minute left. I have one question for you. Sure. Thanks again for the presentation. It was very interesting. I wanted to, you mentioned image compression. I wanted to know about the overhead of the Geo package itself. Is there any associated overhead? And have you made any comparison to MB tiles, for example? There are, there's no difference as far as overhead from using an MB tiles versus a Geo package that we've seen. It's still, if you want to be very fast, just go directly to the tiles table listed in the contents and just start pulling tiles out and referencing them in TMS as you normally would. There might be some calculus you have to do to renumber your tiles based on the way the specs written. However, that's, that's really it. So it's, it's very fast. The image compression, they're not really there to speed up things up, but really to make things smaller. Okay. Thank you again.
 All right, it's about 11.25 now, so I'll get started.  Hi there, everyone. I'm Steve Lander. I'm a software engineer at RGI, and I'm going  to talk to you about implementing OGC Geo package. OGC Geo package being a new standard  that OGC just came out with recently, and this is for a containing feature, raster data,  meta data, schemas, all kinds of stuff into a SQLite database. So a little bit more about  me. I've worked out of Fairfax, Virginia in the United States. I'm experienced in caching  and storing large raster images, so the work I've been doing for, I would say, the past  three years has been involved with using GDAL quite often to take these large raster images,  these geospatially referenced images, and then cutting them into many pieces. And there  are a lot of benefits for doing that, main benefits being that it greatly increases the  time it takes to actually render that information. So, and the work we've been doing for the  past three years has been implementing that encoding standard. We initially started off  with Python, that progressed into Java, and then from Java, we also wanted to put it in  Android. There's been some others as well, JavaScript, but it hasn't really gone anywhere.  So our actual GitHub projects, if you want to know more, software to aggregate geospatial  data, otherwise known as SwagD, and GeoPackagePython, which was the first naive implementation we  started off with. So for SwagD, we were starting off with Java 1.8, which was kind of aggressive.  If I was to do it again, I'd probably start off maybe with Java 1.7 or Java 1.6, just  because not a lot of people are comfortable with going to the latest version of Java.  Typically, you'll find a lot of people with 1.6 or 1.7 out in the wild, and they're kind  of reluctant to go any further into new stuff. But, you know, there's a lot of benefits for  going to Java 1.8 that we saw that we just decided to dive in and see how it went. So,  if you want to see some other presentations we have going on yesterday, and I'm sure it's  going to be on the website, Nathan France had talked about GeoPackage and how that's  changed the way the governments think about standards, and also tomorrow I'll be presenting  some of the open source build tools that I have incorporated into my projects in order  to help me and help my team, you know, reduce our risk, reduce our errors. So, what is the  GeoPackage Encoding Standard? And we're also going to be talking about the actual implementation  we've made and our approach to implementing the encoding standard and what we learned  building the reference implementation. And the lessons learned from this experience,  they are numerous. So, GeoPackage Encoding Standard is a set of conventions for storing  the following, and this is all available at the OTC website. So, vector features I mentioned,  a tile matrix set, which is that GeoReference image cut into many pieces, and we do that  at many scales. So, it's like taking a basically a pyramid data set and throwing it into a  SQLite database. There's also a lot of schema attached to that. Describing your tile set,  describing what levels it exists at, how many there are, and a lot of other helpful things  that you don't really get with MB tiles. MB tiles just kind of dumps a lot of tiles into  a single SQLite database. MB tiles likes to throw some rigor around that so that you  have more information, more metadata. So, yeah, speaking of metadata, you know, you  can describe your, the type of data you have more. Maybe you want to talk about a classification  or, you know, the sky's the limit. And then, extension. So, the extension says, I think  is one of the more powerful features of it. Extension will allow you to do things that  the current standard, the way it's written now, will not let you. So, things like perhaps  you want to add a network data set into your GeoPact. Perhaps you want to add time stamping  to your images, to your tiles. Because tiles can update, tiles can change, and you want  to figure out which ones are the freshest or the most stale, for instance. So, when we  started, and this is something that Vladimir actually had too, if you saw Vladimir Agafonkin's  presentation. So, how do these, why do we have another standard for doing this? I think  it's a little bit better than the situation here is saying because I saw that MB tiles  was pretty good. MB tiles was a really good way to take this idea of a cache map and throw  it onto a mobile device or any device as a single file. And I thought there was a lot  of, a lot of good value there. But also, you don't have to take the entirety of GeoPact.  You don't have to put rectors in there if you don't want to. You don't have to put rasters  in there if you don't want to. You can make a blank one if you really desire to. But I  think that there's nothing wrong with more standards, but it's going to really depend  on your use case. So, what kind of existing technology is already out there that perhaps  could compete with GeoPact or does something similar to what GeoPact does? The main one  you'll see on OGC site is the shapefile. So, shapefile, you'll have your vector information,  you'll have your metadata, but the shapefile is not going to do your raster tile data sets.  KML would kind of fit that need a little bit better than shapefile. You would actually be  able to take your tile map data sets in KML and view them, and also vector as well. But  you don't get some of the nicer parts of the scheme of the metadata and the extensions  with KML. GeoJSON, very fast, very good for vector. And then NPTiles, you could probably  throw vector information into an NPTiles, but the problem there being, as a spec, NPTiles  only supports Web Mercator. Whereas GeoPact, you can support any EPSG code or anything  specific, even your own custom WKB or WKT if you so want it. So, there are more implementers  of GeoPact and actually growing than just ours. GDAL had that quite a while ago for  just features in 111.0. We actually use 111 in our software now. And raster tiles have  just emerged in 2.0. We haven't looked at that. I haven't looked at that personally,  but it's pretty exciting. GeoServer, I talked to some of the GeoServer people that are here.  There's a community supported plugin. That's not everybody's radar, but nothing actually  official yet, as far as I know. Please correct me if I'm wrong. So, Spatial Light, they just  added the version. I checked that probably a month ago, and that was just pretty recent.  Esri has had feature support, I would say, and their supported GeoPact is actually still  ongoing. Raster support is coming, however, you know, the way they deal with tile image,  tiles and tile in a cache is actually fundamentally different from your TMS or your WMTS. They  actually have a binary format, tile cache type situation that doesn't make it a really  quick fit. And then Digital Globe, they have the ability currently to push out the imagery  that you see to a GeoPackage. You can actually export that. And the asterisk is there because  that was code that we actually made. That's the Python naive implementation. So, if you  have a Digital Globe account, you can go there and instead of getting a file folder structure  full of tiles, you can actually just get a GeoPackage, have a SQLite database. And if  your viewer supports it, you can see that. So, our approach. So, what I really wanted  to do was create an API for other developers and for other implementers that they could  just pick and choose what they wanted. So, I knew that the intended user that we were  raking the software for wasn't going to need everything in a single project, right? You  would have certain individuals that maybe would want to deploy this on a server. They  would want to perhaps have users come in and they would have a tile back end on their server  that they would be able to draw from, take those tiles or take that imagery, tile it  there even and then push it out to a user that they could have a downloadable file from.  Or maybe they want to view it on a mobile and tiling just doesn't make sense. And I  wanted to build a UI around that API as just a demonstration so that you could see the  types of things it would allow you to do if you use it on a desktop, if you use it on  a server or if you use it on a mobile device or whatever. And I mentioned we started off  with Java 1.8 and which is really great. Everybody on the team loves it but there are some  problems. There are some features that especially for Android and we'll talk about that later  just aren't going to work in and we had to backport some things. But that's how it goes.  So the core API, we have a common library that shows your typical geospatial things.  You have a data store library which is going to have the classes that allow you to manipulate  a store of tiles. A store of tiles could be a file folder structure of TMS organized images  or it could be a GIF package or it could be an MB tiles. Whatever you have that's storing  tile information, cache map information, that's that library is going to allow you to interact  with those. A GDALT2Tiles project as well exists in there and the name taken from the  Python script by the same name and a lot of the code in that project closely follows  what they're doing and that's basically just taking any GDALT supported image and tiling  it for you and giving you output. The GeoPackage class is typically what people are going to  want. That's going to allow you to verify a GeoPackage that's going to allow you to  open one, to create one, to put imagery into one and then our network extension is actually  an extension that we talked about where you're able to add a network dataset. You have to  add for instance nodes and edges to a GeoPackage and interact and query that information on  your actual device like maybe a mobile application or your laptop using things like ASTAR or  Dykstra routing. We just had a keynote about open software. Interoperability is definitely  something that I have to worry about constantly. I wanted to have that thought about before  coming in and making everything. We had a verification component there that when you  open the GeoPackage, it's going to check literally everything. All 200 pages of that GeoPackage  encoding document, it's going to go through and see if you have been adhering to that.  And then that was so popular that we actually had to make that a separate application so  that other people in the wild, other implementers, other companies could take that and say, hey,  I want to make my GeoPackage interoperable. Do you have anything that could help? And  we're like, sure, this is this verification app. It will help you out and it will actually  give you pointers to the actual specification headline so that you can say, okay, I'm not  following the spec here and what do I do? What does this even mean? Another way we show  for interoperability support was just using the JVM. I probably could have gone Python,  I could have gone Haskell, I could have gone crazy, but the reality is the JVM has allowed,  my development team to use Windows Linux, I personally use Linux to do my development  on, has allowed us to push to Android. So that was very, very important that we have  something that was easily portable. And the outputs that it supports are if you've ever  made a leaflet-based map or if you've ever made a leaflet map, those are the big three  that you'll have support for baked in. Three or five, seven more Mercator, global geodetic,  eco-retangular, and then global Mercator, which you typically don't see a whole lot  of people making global Mercator, but they give it as web Mercator only, not as egregious  in the, in the sort of the distortion that it introduces. So we made a reference to  UI that I mentioned. It's swing-based, which really makes me sad because I don't like swing  at all. I really wish it was kind of an NBC, MVVM situation. But, you know, we're getting  there. Java FX just came out for 1-8, so that's something that's very exciting for the team.  JMapViewer is a, is a jar that I, that I happened to find through some research, and JMapViewer  actually supported tile map sets out of the box, so I didn't have to read write anything.  The problem is it was the only one that was out there. So there are some things that it  does poorly that I wish it didn't do. And, and if we ever moved to a two Java FX there  in that last headline, that JMapViewer would be going away because it's, it's not very,  not very flexible, 3-8, 5-7 only. You know, some, some odd behavior. This is our reference  UI here that I made. Simplicity. It was definitely the goal, three workflows, tiling data, packaging  that data and then viewing it. What we're seeing here is the viewing. And then this is  an Alaska data set pulled from, from a geo package. And you're actually viewing the tile  map imagery there. Very, very basic, very plain. But when you have something like that  available on, on like a mobile device that you don't have to worry about connectivity,  it could be, you know, quite powerful. These are the, some of the inputs we take when we  are doing tiling. For instance, you know, full, full reference system support incoming  and then outgoing. We, we limit that. You'll see the drop down box there. You know, we,  we limit that because of what we know a, a user, what we, what we assume a user will  be able to, to want what they will want. You know, 3-8, 5-7, typically in 90% of the cases.  And then, you know, 3-8, 5, or 4-3, 2-6, kind of, you know, tacit support right now that  will be coming of, you know, actual rigorous support for that will be coming a little bit  later. I'm making, you can also tile directly to a GA package. If you just want to skip  the whole creating, creating images, you can do that and just tile directly to a GA package.  Or if you want to have the flexibility of having all those loose tiles somewhere on  your system, you can, you can do that as well. If you want to change what you're making,  if you don't want to make a huge PNG collection, you want to just make a JPEG collection with  some, with some compression, you can do that as well. And that's a metadata here at the  bottom and where you want to save it. So, then when we actually package, you know, pretty  similar, you can actually change the imagery that you have as well when you're packaging.  You can take a PNG image because you want to save that transparency information and  then convert it to a JPEG because maybe you don't, maybe space is more important, right?  If you're targeting a mobile device, the difference between a GA package that is, that is PNG  and JPEG can be, you know, many hundreds of megabytes.  I mentioned the verifier, verifier is Java FX and a lot better, I think, of a UI than,  than the swing application you just saw, unfortunately. But, you know, redoing an  application, redoing the UI for the application that we have that you just saw is, is a large  task. So, it tests for the following parts of the  GA package encoding standard, all the, all the big ones that you have, the one that's  noticeably absent there is features and features is just because we haven't gotten there yet.  So, the idea of what it looks like, Alaska, Alaska 2.geopackage, which you just saw, was  good. You can, you know, collapse that. You can see the other, other GA package and what  it's past. You can also see, and I apologize, the radiant  blue is the only GA package that I had on my system that would show a warning. But I  think this is a good illustration in that you have a warning here, not necessarily a  breakage. So, for instance, on this line here, it says, this is pulled directly from the  spec. A GA package shall contain, you know, 047503130 GP10 in that scale. That really  doesn't mean a whole lot to a layman, right? But, but all that's saying is, if you have  this one byte header set in the binary file, it should be there, but it's not, you can  still open it. It's not fatal. So, what did we learn building this reference, reference  implementation? What have we learned about coding? What have we learned about ourselves?  So, for Java, I realized really early on that, man, there are just very few choices for viewing  tile data sets. I mentioned JMapViewer. That was really it. And finding that was on some,  some Arcane website that I found that had a download link and I wasn't sure if it was  malware or not, but it's not malware. So, I wish, I really wish we knew more about  Java FX. I actually found out about Java FX from the last Phosphor G in San Francisco  in Berlin and once I found out, I was like, wow, that's amazing. I wish, I wish really  earlier heard about that sooner. A lot of many tiers could have been averted. The Python  implementation that we have for tiling is still far in a way, the fastest way to make  tiles. We researched in Java, you know, multi-threading, multi-processing. How can we, how can we  speed this up? The Python implementation takes, uses the Python multi-processing library and  it is so fast that it will kill your system. You'll use every single core and it won't  be usable, but man, you'll have tiles really quickly. We can't get that with Java right  now, unfortunately. That's because of the JVM, the limitations you have with the JVM.  And you'll find this a lot when you, this last point, you'll find this a lot when you're  dealing with, with database operations. Optimizing the way you interact with the database is  typically giving you your biggest gains in performance. So reduce the amount of SQL calls  you make and then reduce the amount of times you're going to your, your SQLite database  or your database in particular because that's, that's going to take the longest time, right?  Worrying about pieces of code in, in certain areas where you're maybe using a list versus  a hash map, that's not going to get you the type of improvements you'll get versus going  over a database. What have you learned about Android? Compatibility for Android for just  a while. See, I was on the impression like, oh, you have Java, just throw it right on  the Android, on the Android device and it'll just work. But that's not the case. It's still  stuck in 1.6 officially. 1.7 support is, is there. I believe you can make a 1.7 Java  project for Android and just bring it over and it, and it should work. But we didn't  want to take that chance. So the fact that we had Java 1.8 and we had to go back for  a couple, just a couple libraries to do the, the compatibility for Android, that really  stunk. So, so the desktop development for, for Android is really ill-suited. The problem  is when you're developing on a desktop for interacting with a SQLite database, those  libraries and those, that functionality that you'll have available to you is going to differ  than it is on an Android device. Android device having their own SQLite implementation that's  kind of baked into Android, whereas if you're on the desktop, you have to, you have to bring  that in. That's not something that you just get with Java. And then, well, big one, you  know, a buffered image and a desktop Java versus a bitmap image on an, on an Android  Java implementation. So you kind of have to have an adapter there to, to help you bridge  that. And, and they're not necessarily very compatible. And then, the different testing  frameworks that, that exist on, on, you know, JUnit versus an, an Android one on Android.  And then, there was frustratingly no, no globally available, most current Android jar available  like Maven. So we just had to, had to throw that in a source and reference it. So regarding  the spec, you know, I, I thought I'd been working on this for three years. There's, there are  still questions about the spec, still, still different interpretations on a single word.  And that's kind of frustrating. Even though they tried to be as verbose and as precise  as possible, you still have differing opinions and, and the ways that you can, you can do  your, you can do your tileset. So compromises are the key. And I already mentioned, Lingokin  differ. So in general, think about who you want to use your code and, and how, and then  constrain that list. You saw that the UI had, it was actually gave you very, very few options.  So, all right. Thank you for listening.  Thank you, Steve. Do you have any questions for, for Steve?  One of the problems with extensions is as soon as you start using extensions, you reduce  who you can share this data with because extensions are not universally done. So.  Sure. So the good part about extensions is if you're, if you're making a Geo package  implementation, you will just ask the Geo package, what, what extensions do you have?  And if it has an extension that you don't have, you still, you can still open it. You  can still get your, your functionality that you want. You don't, you're not losing anything.  You just, you know, if you get something with an extension that you don't support,  you can still look at the tiles. You can still look at the stuff that you can still get all  the things you get with a vanilla Geo package. True. If you, if you have an extension, you,  that's going to be implementation specific. And that's, that's kind of a, that's kind  of a shame. But hopefully you're registering your extension with OGC or you're registering  it with the body so that you can, you can have people and you know, tie it back to the  keynote. You know, you got to be as open as possible. And, and that'll help. So you have  to mitigate, unfortunately. Sure. Sir?  What was your use case for, for using Geo? Well, you are, what, what, you're only using  in your application like we did on the online, just for the, on the third?  The use case was we, we wanted fast base maps and fast data on a mobile device. So there,  there were, you know, you could have MB tiles. MB tiles was great. MB tiles could work. However,  more information we felt was better. The more rigor you throw around it. So that's why we  went with Geo package. And third.  Hmm? And third.  Yeah. Yeah. Sir?  Do you think that's the best use case for Geo package or is it better as kind of a universal  standard for data interchange rather than a mobile device?  It's a, that's a good question. I think that Geo package should exist outside of, outside  of the mobile realm. I think the mobile realm is just one in which that it works particularly  well. There are plenty that don't work very well. So for instance, storing your data,  storing your tile map data on the server in Geo package, I would not recommend that because  you will have problems with SQLite at a certain point. Eventually you'll get to a size, I  mean, in the size can be terabytes of for SQLite, right? But eventually you get to a point where  inserts no longer make sense. Working with that Geo package no longer makes sense. You might as well  just have a big binary blob of tiles or file folder structure or something similar.  Any others? Other question? I think we have one minute left. I have one question for you.  Sure.  Thanks again for the presentation. It was very interesting. I wanted to, you mentioned  image compression. I wanted to know about the overhead of the Geo package itself. Is there  any associated overhead? And have you made any comparison to MB tiles, for example?  There are, there's no difference as far as overhead from using an MB tiles versus a Geo  package that we've seen. It's still, if you want to be very fast, just go directly to the  tiles table listed in the contents and just start pulling tiles out and referencing them  in TMS as you normally would. There might be some calculus you have to do to renumber  your tiles based on the way the specs written. However, that's, that's really it. So it's,  it's very fast. The image compression, they're not really there to speed up things up, but  really to make things smaller.  Okay. Thank you again.
PT4-08_0_95.png PT4-08_96_755.png PT4-08_756_2111.png PT4-08_2112_3251.png PT4-08_3252_3923.png PT4-08_3924_4379.png PT4-08_4380_6455.png PT4-08_6456_7931.png PT4-08_7932_9455.png PT4-08_9456_12191.png PT4-08_12192_12287.png PT4-08_12288_14219.png PT4-08_14220_16271.png PT4-08_16272_19391.png PT4-08_19392_20639.png PT4-08_20640_21359.png PT4-08_21360_22907.png PT4-08_22908_23567.png PT4-08_23568_24347.png PT4-08_24348_24659.png PT4-08_24660_25511.png PT4-08_25512_25763.png PT4-08_25764_28499.png PT4-08_28500_30995.png PT4-08_30996_31799.png PT4-08_31800_32123.png PT4-08_32124_32267.png PT4-08_32268_32339.png PT4-08_32340_37955.png PT4-08_37956_38048.png