Using and extending GeoPackages
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kalberer, Pirmin (Sourcepole)
GeoPackage is an OGC standard combining vector data and raster tiles in a single database file. Since its official publication in February 2014 adoption has grown very quickly and the new format got support by most major Open Source and proprietary software products. Its combination of easy distribution and efficient use thanks to an SQL interface makes it interesting in a wide range of usage scenarios from embedded devices up to Open Data portals. This presentations shows the current state of support in Open Source applications like GDAL and QGIS, but also covers recent and planned extensions of the format. Examples are the proposed GeoPackage Elevation Extension, an extension for storing point cloud data and the recent QGIS all-in-one GeoPackage format. The latter implements storing map styling information right in the GeoPackage file, allowing to distribute a dataset together with map views and print layouts in one file.
 But now we will hear from Köln and Kaibaba from the source for company in Switzerland about using and extending the due. So you hear me? Yes. I'm talking about Q-Ages and that's the first talk about Q-Ages where I talk about using, reusing it and not only proposing it in the format. I started making advertisement for this format before it exists. First time in Barcelona, on the page of LIOND and after that words became this OTC standard and now I'm talking about using it. Our company is mainly involved with Q-Ages development so I concentrate a little bit on Q-Ages. I don't think you might propose more than you do. You can try the handheld. Can you start? I'll do it. You can make it do it. I'll do it. Geo package is an OTC standard which is quite young. It was published finally in 2014. It's written in the standard, it's a good description. It's an open standard for a platform independent container for distributing and using geospatial data. So that's two things together. You can directly use it and you can easily distribute it. It's mainly vector and raster data which you can store in SQLite DB container. So you can think of it like an spatial extension to SQLite. So similar to PostJS and Postgres. It has spatial data types, it adds spatial data types and functions to SQLite for vector data and also for raster data where you can store tile permit data sets. The feature data is according to OTC simple feature types. It's stored as WKB. You have points, lines, polygons with M and ZM geometries. So that's the basic geometries. It's WKB and there is one restriction. There is only one geometry column per table. You have indexes and as I said you have raster tile sets. That's the format and now I'm talking about using that format for the open source geospatial world. GDAL OGR is the most important implementation and it has added vector features from GeoPackage in 111 and in GDAL 2.0 where there was a convergence between raster and vectors. There has been the raster support added and you can have now both through GDAL. Then QtjS is using GDAL so it mainly supports everything GDAL supports. So that's also important when you install QtjS, it depends on the version of GDAL which is installed. You can see that in the BAU dialog and it's important to have 2.0 or 2.1. Geotools from the Java world does support GeoPackage as well. I have my notes. They started with version 11, they added tiles and feature support and in the meantime they also added support for the R trees. GeoServer has also good support for GeoPackage. They read GeoPackages, raster and vectors and they also export GeoPackage. So when you have a WFS you can also ask for GeoPackage formats in WMS and WFS and also WPS. It's an additional module doing that. On the proper diary site we have FME which is important for many people which has also support for reading vectors from GeoPackages reading and writing. And they plan to support rosters. RGS supports, there is RGS for desktop which supports both in the meantime. I have the versions somewhere written down. From Nintendo 2 they added GeoPackage support, reading and writing for vector features and 10.3 they added tiles, reading and writing. And RGS Pro supports reading at least GeoPackage features. And another proper diary one map info also supports vector features. So this is quite good for a young standard, two years after the standard was published, really all major GIS supports this format. Not all of them the same level but it's a good beginning. So now I go to QTS, there you can easily load GeoPackages, you can load it in the browser which is embedded. You can drag and drop GeoPackages into the canvas or into QTS. And you can also add vectors and raster layers with the file dialog. Since 2016 you can create new GeoPackages. I show that shortly after. And with that you can digitizing and save GeoPackages. There are some limitations, the embedded DB browser does not support everything, it does support for other formats. Especially Postgres and also Spatialite. And column editing is also not fully supported which is also more difficult with SQLite. It's not so easy to rename columns and add columns and remove them. It's not so easy like in PostGS where you can alter table, rename column. In Spatialite you have to do more for that. But it's planned. And also difference to Spatialite is that you can't store styles yet. That's not so well known functionality of QTS, you can store your layer styles into PostGS and also into Spatialite. But not in GeoPackage yet. In GeoPackage they have support for extensions which is an important thing for me. It's written in the spec how you declare or define extensions and there are already a few of them. The first one is in the spec itself, it's support for other geometry types than the simple features. So you have arcs support with this nonlinear geometry types extension. It's only a question of declaration and then it's a valid GeoPackage. And GDAL defined the extension for non-spatial data which may be surprisingly not supported or not officially supported by GeoPackages. But it's just a matter of adding metadata and doing a proper specification. And this has been done by GDAL. There's also support for point cloud data, to store point cloud data in a GeoPackage. There I have a link to a presentation. And I was also informed that there are extensions from the NGA. One is this feature tile link extension. This adds support for adding relations between features and tiles. And the geometry index is similar to the R3. But there are obviously platforms like Android or iPhone where they have old S-colite implementations without R3 support. So that's why they defined this geometry index extension. And there is another one which is for elevation data that's from the OTC itself. There was a call for testing that so it's still not official. But it's also a good example of an extension, how to declare an extension. And it's about storing tips with elevation information in the GeoPackage. And the next thing is an extension I wrote. Although as an example how to define extensions. It is about writing a QJS project, all the mapping information and styling information directly in the GeoPackage file. Which means you can distribute the GeoPackage with included maps. So you have to also include some graphics like logos in a print layout or SVG markers which aren't in the standard installation. And this is a QJS plugin, it is very new, still experimental and for exchanging data together with a map. So that's time to switch to QJS. First I show you a little bit how to work with GeoPackages. I have some GeoPackages with me. For instance some imagery. I drop it, the extent is a little bit wrong. So that's a collection of imagery we use in a project. And so that's raster data and you can also add vector data. And then it asks somewhere on the screen. What you want to add, let's take this. So these are polygons. And so on. What you also can do if you have other data sources, you can go here and save the layer. And you have the selection GeoPackage. So if you have, let's say, a shape file layer, you can save that as a GeoPackage. There is one limitation which is hurting a little bit. You can't add an additional layer to an existing GeoPackage here. I think that's the most important features for working with GeoPackages. And then I quickly show the plugin. So I load a GeoPackage again. Let's say, oh no, one thing I forgot. Here I have also the possibility to start a new GeoPackage. New GeoPackage layer. And then it creates a GeoPackage file database. And you can add tables with fields and so on. And then you can start editing, saving as with other data sources. And I want to load... I want to add a vector layer. One more thing, when you have a mixed GeoPackage with raster and vectors, then there is a problem with drag and drop because it takes the provider which comes first. It doesn't ask you which one you want to import. And that's the raster provider. So it imports the raster. So if you have both, then you have to add the vector layer with the vector dialog and select the file. Okay. So now that's my example to store a project into this GeoPackage. I style that a little bit. Okay. And then I save this map in the GeoPackage I have the data from. And that's this icon here, write the project in the GeoPackage. And now it's written in the GeoPackage. And to test it, start from scratch and load this GeoPackage again. And now it looks the same. So the project is saved. And also if I add a print composer, this is also saved together with an image. And, yeah. So the last thing, the library for that. I wrote it as a library. And then I also have a GeoPackage which can also use from command line. So you can add a project from command line. And also show what's included in a package. So you can call this info with the GeoPackage and you get the information what's inside this GeoPackage. And this library is used in the Qtjust plugin for having this functionality. So to finish up, that's the GeoPackage advantages. You can exchange data and use it directly. Single file, vector and roster from mobile to server with extensions. So I ask you to promote your package, ask for GeoPackage, or if you have data, provide GeoPackages. It's a really useful format. Thanks. Yeah, thank you, Perman, for this great talk. Are there any questions? Yeah, thanks. How does this relate to Spatialite and Rastolite, which are also spatial extensions on SQLite? Should you see the GeoPackage as the successor of them, or do they have both their use cases? For me, we should see GeoPackage as a successor. I mean, right now, Spatialite has more functionality, but GeoPackage is standard supported by everyone. So we should go to GeoPackage in the future. Further questions? Can you drag a new layer using your browser into an existing GeoPackage? That would be easier than using a command. Adding a Qtious layer to an existing GeoPackage. And the browser, no, you can't. It's not supported yet. But you have some projects to add to the missing functionality, but it's good to know what is missing. That's really not that easy. Any further questions for Perlman? If you add a new GeoPackage layer in Qtious, could you add it to existing GeoPackage file? Or is it a new file every time you add a new GeoPackage layer? I'm not 100% sure whether it's the same question. Then the answer is no. You can't add a second Qtious layer to an existing GeoPackage, but you can do that on the command line easily with adding an additional layer. So you have to do it right now. Or was it the other way around? Or you want to add multiple layers from one GeoPackage into a Qtious, adding as Qtious layers? No, I think it was that. And I see the demand for that is really important. So we would have more time. Thank you for the talk. Currently we are providing lots of raster data as a GeoTiff tile. Do you think it would be worse migrating to GeoPackage because I saw your file were quite huge and I guess because of the indexing, it's quite fairly fast to access the information. I don't think that it's generally faster than files. It makes a difference that you have only one file, but sometimes this is also disadvantage. I mean, this one file can get huge. So I think under some circumstances you want to have multiple files and then it's okay. But I mean it's only a file system limit for having these rasters in one GeoPackage. It's a real problem. It depends on your file system. Maybe the limitation is 4 gigabytes and that could be a few. The indexing allows you to access quicker the different images. I don't know exactly how it's organized in GeoPackage, but it was my assumption. Maybe I'm wrong. I think you are trying to index for accessing the different images or part of the images depending on how you organize them. I only know the R3 indexes for vector data and I'm not sure whether they are also used for raster data, but to also have these pyramids and so on. I wouldn't expect that it's faster than on T-files. Okay, thank you. So, the session would be time to switch rooms, but we could have more questions if there are any still open. As said, the second talk was cancelled, unfortunately, so the session would be then over. Thank you, Perman, again for the start.
 But now we will hear from Köln and Kaibaba from the source for company in Switzerland  about using and extending the due.  So you hear me?  Yes.  I'm talking about Q-Ages and that's the first talk about Q-Ages where I talk about using,  reusing it and not only proposing it in the format.  I started making advertisement for this format before it exists.  First time in Barcelona, on the page of LIOND and after that words became this OTC standard  and now I'm talking about using it.  Our company is mainly involved with Q-Ages development so I concentrate a little bit on Q-Ages.  I don't think you might propose more than you do.  You can try the handheld.  Can you start?  I'll do it.  You can make it do it.  I'll do it.  Geo package is an OTC standard which is quite young.  It was published finally in 2014.  It's written in the standard, it's a good description.  It's an open standard for a platform independent container for distributing and using geospatial data.  So that's two things together.  You can directly use it and you can easily distribute it.  It's mainly vector and raster data which you can store in SQLite DB container.  So you can think of it like an spatial extension to SQLite.  So similar to PostJS and Postgres.  It has spatial data types, it adds spatial data types and functions to SQLite for vector data  and also for raster data where you can store tile permit data sets.  The feature data is according to OTC simple feature types.  It's stored as WKB.  You have points, lines, polygons with M and ZM geometries.  So that's the basic geometries.  It's WKB and there is one restriction.  There is only one geometry column per table.  You have indexes and as I said you have raster tile sets.  That's the format and now I'm talking about using that format for the open source geospatial world.  GDAL OGR is the most important implementation and it has added vector features from GeoPackage in 111  and in GDAL 2.0 where there was a convergence between raster and vectors.  There has been the raster support added and you can have now both through GDAL.  Then QtjS is using GDAL so it mainly supports everything GDAL supports.  So that's also important when you install QtjS, it depends on the version of GDAL which is installed.  You can see that in the BAU dialog and it's important to have 2.0 or 2.1.  Geotools from the Java world does support GeoPackage as well.  I have my notes.  They started with version 11, they added tiles and feature support and in the meantime they also added support for the R trees.  GeoServer has also good support for GeoPackage.  They read GeoPackages, raster and vectors and they also export GeoPackage.  So when you have a WFS you can also ask for GeoPackage formats in WMS and WFS and also WPS.  It's an additional module doing that.  On the proper diary site we have FME which is important for many people which has also support for reading vectors from GeoPackages reading and writing.  And they plan to support rosters.  RGS supports, there is RGS for desktop which supports both in the meantime.  I have the versions somewhere written down.  From Nintendo 2 they added GeoPackage support, reading and writing for vector features and 10.3 they added tiles, reading and writing.  And RGS Pro supports reading at least GeoPackage features.  And another proper diary one map info also supports vector features.  So this is quite good for a young standard, two years after the standard was published, really all major GIS supports this format.  Not all of them the same level but it's a good beginning.  So now I go to QTS, there you can easily load GeoPackages, you can load it in the browser which is embedded.  You can drag and drop GeoPackages into the canvas or into QTS.  And you can also add vectors and raster layers with the file dialog.  Since 2016 you can create new GeoPackages.  I show that shortly after.  And with that you can digitizing and save GeoPackages.  There are some limitations, the embedded DB browser does not support everything, it does support for other formats.  Especially Postgres and also Spatialite.  And column editing is also not fully supported which is also more difficult with SQLite.  It's not so easy to rename columns and add columns and remove them.  It's not so easy like in PostGS where you can alter table, rename column.  In Spatialite you have to do more for that.  But it's planned.  And also difference to Spatialite is that you can't store styles yet.  That's not so well known functionality of QTS, you can store your layer styles into PostGS and also into Spatialite.  But not in GeoPackage yet.  In GeoPackage they have support for extensions which is an important thing for me.  It's written in the spec how you declare or define extensions and there are already a few of them.  The first one is in the spec itself, it's support for other geometry types than the simple features.  So you have arcs support with this nonlinear geometry types extension.  It's only a question of declaration and then it's a valid GeoPackage.  And GDAL defined the extension for non-spatial data which may be surprisingly not supported or not officially supported by GeoPackages.  But it's just a matter of adding metadata and doing a proper specification.  And this has been done by GDAL.  There's also support for point cloud data, to store point cloud data in a GeoPackage.  There I have a link to a presentation.  And I was also informed that there are extensions from the NGA.  One is this feature tile link extension.  This adds support for adding relations between features and tiles.  And the geometry index is similar to the R3.  But there are obviously platforms like Android or iPhone where they have old S-colite implementations without R3 support.  So that's why they defined this geometry index extension.  And there is another one which is for elevation data that's from the OTC itself.  There was a call for testing that so it's still not official.  But it's also a good example of an extension, how to declare an extension.  And it's about storing tips with elevation information in the GeoPackage.  And the next thing is an extension I wrote.  Although as an example how to define extensions.  It is about writing a QJS project, all the mapping information and styling information directly in the GeoPackage file.  Which means you can distribute the GeoPackage with included maps.  So you have to also include some graphics like logos in a print layout or SVG markers which aren't in the standard installation.  And this is a QJS plugin, it is very new, still experimental and for exchanging data together with a map.  So that's time to switch to QJS.  First I show you a little bit how to work with GeoPackages.  I have some GeoPackages with me.  For instance some imagery.  I drop it, the extent is a little bit wrong.  So that's a collection of imagery we use in a project.  And so that's raster data and you can also add vector data.  And then it asks somewhere on the screen.  What you want to add, let's take this.  So these are polygons.  And so on.  What you also can do if you have other data sources, you can go here and save the layer.  And you have the selection GeoPackage.  So if you have, let's say, a shape file layer, you can save that as a GeoPackage.  There is one limitation which is hurting a little bit.  You can't add an additional layer to an existing GeoPackage here.  I think that's the most important features for working with GeoPackages.  And then I quickly show the plugin.  So I load a GeoPackage again.  Let's say, oh no, one thing I forgot.  Here I have also the possibility to start a new GeoPackage.  New GeoPackage layer. And then it creates a GeoPackage file database.  And you can add tables with fields and so on.  And then you can start editing, saving as with other data sources.  And I want to load...  I want to add a vector layer.  One more thing, when you have a mixed GeoPackage with raster and vectors,  then there is a problem with drag and drop because it takes the provider which comes first.  It doesn't ask you which one you want to import.  And that's the raster provider. So it imports the raster.  So if you have both, then you have to add the vector layer with the vector dialog and select the file.  Okay.  So now that's my example to store a project into this GeoPackage.  I style that a little bit.  Okay.  And then I save this map in the GeoPackage I have the data from.  And that's this icon here, write the project in the GeoPackage.  And now it's written in the GeoPackage.  And to test it, start from scratch and load this GeoPackage again.  And now it looks the same.  So the project is saved.  And also if I add a print composer, this is also saved together with an image.  And, yeah.  So the last thing, the library for that.  I wrote it as a library.  And then I also have a GeoPackage which can also use from command line.  So you can add a project from command line.  And also show what's included in a package.  So you can call this info with the GeoPackage and you get the information what's inside this GeoPackage.  And this library is used in the Qtjust plugin for having this functionality.  So to finish up, that's the GeoPackage advantages.  You can exchange data and use it directly.  Single file, vector and roster from mobile to server with extensions.  So I ask you to promote your package, ask for GeoPackage, or if you have data, provide GeoPackages.  It's a really useful format.  Thanks.  Yeah, thank you, Perman, for this great talk.  Are there any questions?  Yeah, thanks.  How does this relate to Spatialite and Rastolite, which are also spatial extensions on SQLite?  Should you see the GeoPackage as the successor of them, or do they have both their use cases?  For me, we should see GeoPackage as a successor.  I mean, right now, Spatialite has more functionality, but GeoPackage is standard supported by everyone.  So we should go to GeoPackage in the future.  Further questions?  Can you drag a new layer using your browser into an existing GeoPackage?  That would be easier than using a command.  Adding a Qtious layer to an existing GeoPackage.  And the browser, no, you can't.  It's not supported yet.  But you have some projects to add to the missing functionality, but it's good to know what is missing.  That's really not that easy.  Any further questions for Perlman?  If you add a new GeoPackage layer in Qtious, could you add it to existing GeoPackage file?  Or is it a new file every time you add a new GeoPackage layer?  I'm not 100% sure whether it's the same question.  Then the answer is no.  You can't add a second Qtious layer to an existing GeoPackage, but you can do that on the command line easily with adding an additional layer.  So you have to do it right now.  Or was it the other way around?  Or you want to add multiple layers from one GeoPackage into a Qtious, adding as Qtious layers?  No, I think it was that.  And I see the demand for that is really important.  So we would have more time.  Thank you for the talk. Currently we are providing lots of raster data as a GeoTiff tile.  Do you think it would be worse migrating to GeoPackage because I saw your file were quite huge and I guess because of the indexing, it's quite fairly fast to access the information.  I don't think that it's generally faster than files.  It makes a difference that you have only one file, but sometimes this is also disadvantage.  I mean, this one file can get huge.  So I think under some circumstances you want to have multiple files and then it's okay.  But I mean it's only a file system limit for having these rasters in one GeoPackage.  It's a real problem. It depends on your file system.  Maybe the limitation is 4 gigabytes and that could be a few.  The indexing allows you to access quicker the different images.  I don't know exactly how it's organized in GeoPackage, but it was my assumption.  Maybe I'm wrong.  I think you are trying to index for accessing the different images or part of the images depending on how you organize them.  I only know the R3 indexes for vector data and I'm not sure whether they are also used for raster data, but to also have these pyramids and so on.  I wouldn't expect that it's faster than on T-files.  Okay, thank you.  So, the session would be time to switch rooms, but we could have more questions if there are any still open.  As said, the second talk was cancelled, unfortunately, so the session would be then over.  Thank you, Perman, again for the start.
foss4g-2016-1125-using_and_extending_geopackages-hd_0_563.png foss4g-2016-1125-using_and_extending_geopackages-hd_564_671.png foss4g-2016-1125-using_and_extending_geopackages-hd_672_779.png foss4g-2016-1125-using_and_extending_geopackages-hd_780_959.png foss4g-2016-1125-using_and_extending_geopackages-hd_960_16835.png foss4g-2016-1125-using_and_extending_geopackages-hd_16836_23111.png foss4g-2016-1125-using_and_extending_geopackages-hd_23112_27827.png foss4g-2016-1125-using_and_extending_geopackages-hd_27828_29879.png foss4g-2016-1125-using_and_extending_geopackages-hd_29880_30611.png foss4g-2016-1125-using_and_extending_geopackages-hd_30612_32699.png foss4g-2016-1125-using_and_extending_geopackages-hd_32700_33695.png foss4g-2016-1125-using_and_extending_geopackages-hd_33696_33743.png foss4g-2016-1125-using_and_extending_geopackages-hd_33744_33791.png foss4g-2016-1125-using_and_extending_geopackages-hd_33792_33863.png foss4g-2016-1125-using_and_extending_geopackages-hd_33864_34679.png foss4g-2016-1125-using_and_extending_geopackages-hd_34680_34739.png foss4g-2016-1125-using_and_extending_geopackages-hd_34740_35015.png foss4g-2016-1125-using_and_extending_geopackages-hd_35016_35243.png foss4g-2016-1125-using_and_extending_geopackages-hd_35244_37199.png foss4g-2016-1125-using_and_extending_geopackages-hd_37200_37494.png