Past, Present, & Future of MapProxy
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Tonnhofer, Oliver (Omniscale)
More than three years ago MapProxy started as a small tile cache with the ability to serve regular WMS clients. MapProxy grew from that to a powerful and flexible proxy for maps. Features like the security API, the ability to reproject tiles, support for coverages from Shapefiles or PostGIS and the various tools are just a few things that make MapProxy to stand out. MapProxy is used in countless projects - by federal or state agencies and institutions, by universities, students and hobbyists, by small, national and international companies - all around the world. It is used to combine multiple WMS services to one, make WMS servers available in tiled clients or to restict access to georaphic boundaries. This presentation will show you the most important features that were added to MapProxy in the last years. All features will be explained with practical use cases. Topics: - Cascading WMS: combine multiple heterogeneous WMS services to one, with coverages and unified FeatureInfo - Tiling: create Google Maps/OpenStreetMap compatible tile services from WMS services that do not support the web mercator projection - Tiling: reproject tiles from web mercator to a local projection - Security: give users access to single layers, restricted to user-dependent polygons - Render server: directly integrate MapServer or Mapnik into MapProxy - Tools: calculate scales, estimate the number of tiles, read capabilities, re-seed areas, ... This presentation will also be about the future of MapProxy and the road to version 2.0.
 And I'm really happy to introduce you on the topic of the Peace, Wonster, Sounders of Omniscape, something that I've talked to a lot in Germany. And yeah, I think you're presenting something like your baby or whatever you call that. So you're talking about past, present, and future of that talk to you, and I'm sure they only will be in the future. Thank you, Till. And hi, I'm Oliver Tonopoul. I'm the core developer of MapRoxy, and as Till said, I'm one of the co-founders of Omniscale with a small company in the northwest of Germany. And we do a lot of open source and open-sreenap stuff. And today, I want to talk about the past, the present, and the future of MapRoxy. And the past. Well, how we started with MapRoxy, it was at the end of 2008. And we had the idea for MapRoxy. We knew that tile caching is past, and that it will be the future. But we also knew that the government requires WMS services. And our idea was we just combined a tile cache with a full WMS and make a product out of it. And this is how the first version of the MapRoxy looked like as a schema. We have, on one side, we have clients that communicate with MapRoxy, and MapRoxy itself communicates with other WMS servers or tile servers. And the first features of MapRoxy, of course, was that it supported tile requests. So clients make requests to MapRoxy, and MapRoxy delivers tiles. When these tiles are missing, MapRoxy would request the service to fill the cache. We also supported WMS from the first version on, cascaded request. So MapRoxy just forwarded the request for the same exact bounding box and WSNA and projection to the source. And MapRoxy would return that image. But we also supported responding to a WMS request from cached data. And this is quite unique for MapRoxy. And it does it by, maybe you can see here, just a map with these should be tiles. And when a request comes in, MapRoxy just combines multiple tiles that are all affected by that request. We'll create one image, merge the tiles together, scale them to the exact resolution that was requested by the user. And it even supports the reprojection. So when your cache is in one projection, but the user requests another projection, it does it. Yeah, it does the reprojection. It does it outbound, inbound, and on the fly. So when you have a cache in one projection, it can reproject outbound data. But it also can reproject on the inbound when you want to build up a cache in one projection, but your source doesn't support it. So we project it then. And on the fly for cascaded request. And we used MapRoxy in production since 2009. But it was only used by us. And software only gets better when you have more users, when you get more feedback. And when people with other use cases come to you and say, it would be great if MapRoxy could do that. And so we open source MapRoxy. And all the features are listed, where in our first open source release. We started with version 0.8. And it was in May 2010. And from then on to the present, we had now six major releases in the last three and a half years. And we added lots of features, most of the time to make things easier, and also to solve completely new use cases. And now I want to highlight some of the features that we added in the last three and a half years. One feature was with the 0.8.3 release that we now supported, basically any tile source. You can just write down your own URL template. And you can access any XYZ tile source or WMSC that requires a fixed parameter order. We also improved the seeding tool. Seeding is a process of pre-generating the tiles so that your cache is not empty at the beginning. And since then, the seeding tool supports parallel seeding. So MapRoxy makes multiple retress to the same source. And we also support a feature that we call coverages. And with that, you only seed the required areas instead of the complete bounding box. And you can see it in this image. If you want to seed the UK, and you just would use a bounding box, you see that you're seeding a lot of water areas and neighboring countries. And with MapRoxy, you can just load up a polygon and just seed within that polygon. And this makes seeding a lot faster. And you're saving a lot of disk space. And you can load polygons from geojazen files, shapefiles, even databases like postures and Oracle, all thanks to the OGR library. With 0.9, we added a multi-MapRoxy, which is a tool to have multiple MapRoxy configurations run all in the same instance of MapRoxy. So if you want to run different WMS services, you don't have to start up multiple MapRoxy. Since then, we support coverage also for sources. And I will show that later. And we also added a demo service. Since then, you have the ability to easily check all your layers that you've configured with a simple open-layers client. With the 1.0 release, we added a security API where you can write your own functions to define what a user is allowed to do. And I will also show you go into detail with that in a minute. And MapRoxy supports WMS feature info. And you can optionally add XSL scripts to transform the feature information. So if you have two different WMS servers and the format is different of the feature info, you can write your own scripts to unify that format. Then we added WMTS support and MapRoxy Utils, which is a tool that combines multiple subcommands. And one subcommand is the serve develop. And with that command, you can easily start a testing server for MapRoxy. It will check if your MapRoxy configuration changes and will then just reload it. And you don't have to deal with deployment in Apache, et cetera. And we also support, since then, MapNIC and MapServer directly. So you don't have to set up a MapServer inside Apache, for example. With the next releases, we added a few new cache backends. Before that, MapRoxy always stored the tiles on the file system as simple files. And we added MB tiles and cache to be backends. We added more Utils, for example, the Scales tool that allows you to convert between scales and resolutions, which is quite handy when you deal with MapRoxy configuration. With 1.4, one of the larger features was the clipping of repress in the security API. And I will show you that also in a minute. With 1.5, we supported reprojection of tiles on the fly. So you have tiles in one projection, MapRoxy can transform them into another projection. We support mixed image mode, which is useful for aerial images, which you want to store in SJPEC. But at the boundaries where the aerial images are, where you don't have aerial images, where you want to have transparency, MapRoxy will store these tiles as PNG. And even more MapRoxy Utils, like the Grid tool, which shows you how many tiles are there for each zoom level in your grid definition, and even how many tiles are within a coverage, as you defined. 1.6 released last week, I think. We added more cache back ends, and also the Decorate Image API, which allows you to write your own code that can manipulate the images to go to the client. You could add watermarks or something like that. So now we are at the present. And yeah, lots of features. We now had nearly 2,000 commits. The change log lists 260 bullet points. I just highlighted the most important. And so you might ask, this must be a lot of code. But MapRoxy itself is written in Python. Python is a very powerful programming language. And so the code base is relatively small. It's just a bit over 30,000 lines of code. And on top of that, we have nearly the same amount of code for tests, which I think is really great for a software like that. We have a lot of low level tests, and we also have a lot of high level tests, system tests that take a complete MapRoxy configuration. Start up a MapRoxy, make requests to it, check if the response is valid XML document. If everything is valid, and it also checks the request MapRoxy makes to source request. And we have a Jenkins instance, which runs all tests of each commit. It's also integrated into our GitHub repository. So we get a warning when someone makes a pull request that breaks the build. And you see the small red green marker when everything is fine. Help, documentation now is 110G pages when you get that as PDF. We have a mailing list. You can get support. There are now a lot of local open source companies that use MapRoxy that have knowledge in MapRoxy. And of course, we do support for that. So the interesting part is what can you do with it? And now some use cases of MapRoxy. Well, of course, it is a tile case, so you can use it for simple tile caching. It's used in the German broadband Atlas for one example. The nice thing about the tile cache of MapRoxy is, of course, the seeding tool. You can accelerate existing WMS services. This is used a lot in Germany, where we have a lot of old and rusty and slow WMS service. And they want you to accelerate them. As I said, you can transform tiles. And this is used by a few open street map contributors. And they have access to open data tiles from a local tile server. But these tiles are not in the WebMercator projection. And they use MapRoxy to get the data into a jostem, an open street map editor, so that they can digitize roads and buildings. Another example is you have your own WMS server. But unfortunately, that WMS doesn't support the WebMercator projection. But you want to overlay your WMS server with open street map tiles, with Google Maps, et cetera. You can use MapRoxy to re-project the data. Now the security API. MapRoxy can restrict access to your source services. And it does this with a security API. So you can write your own code that connects to your own user database, for example. And your code can decide what the user is allowed to do. You can limit services, like WMS or tile request, the request type, like feature info, and the layers, but also the areas. And for example, you can say, use A is allowed to see layer X, but only within the boundaries of Germany, for example. And everything else is clipped. And in this case, it's made transparent so that you can see the layer below. And this is quite powerful. Satellite provider uses this to restrict the access to paid customers. And it also works with tile request. And last but not least, one larger use case. This is from an actual national network operator in Germany. And their customers, they have a network on the last mile to the customers. And that requires them to have access to official cadastral maps. And in case of Germany, that means Germany, we have 16 federated states. And so we also have 16 cadastral WMS services. So WMS is a standard, so it's quite easy to use it. That's what you would think. But when you look in detail to all these WMS services, you will see that the cadastral services are mostly not free. So you have to have some user credentials. And they all use different authorization methods. Some of the WMS implementations are just broken. They don't return any G codes. They return invalid bounding boxes. They don't behave like the OTC standard says. And there's not a single projection that's handled by all, et cetera. So when you want to make these WMS services available to hundreds of in-house users, then good luck. But you can use MapRoxy for that to unify all services. You can have a single URL for all cadastral maps. And you can use the coverage feature for all your sources. So you can say that this source is only a database data in this region, and when a request comes in for this region, MapRoxy will only make a request to that server and not to the other one. The authorization is hidden from the user. The capabilities are valid. All sources are transformed to a single projection. And we can even make the background transparent for services that do not support that. So make one WMS available to hundreds of in-house users. That's easy. OK, that's with the present and the use cases. I now want to give you a quick look into the future. Now 1.6 is out, so we're now working on 1.7. We don't have any larger features planned, because we already are making plans for the 2.0 release. And we will make some backwards incompatible changes used to refactor some parts. We will also support Python 3. And we will add a new background service. And we will support Python 3. We will skip the 2.5. And new background service. This is the last point I want to make. First, I have to explain the process model of MapRoxy. MapRoxy is a retrace base. The retrace comes in when a threat deals with a process and the response goes out. But as soon as the response is out, you can't do anything anymore, because the next request comes in. And this is given by the web service gateway interface that MapRoxy implements. And it's similar to how CGI and Apache modules and Java setlets work. So limitations are that you don't have any long-running processes. As soon as the request ends, your process stops. And you can't really communicate between requests. There are workarounds. MapRoxy uses, for example, an external tool to have a long-running process to see tiles. And MapRoxy also uses file logs for some synchronization, so that not two processes make the same request to the source. But it's all limited. If we had a long-running background service, we could implement all these features. We could add something like a priority queue, so that we could limit the load to the sources. Now, because one process would now, OK, we have two running queries to the source. We are not making any more. We're just queuing that. We could add seeding without having cron task, et cetera. We could rate limit request and implement adaptive pre-seeding, et cetera. So a lot of features that would be possible with that. And we have a MapRoxy render daemon that's available at GitHub. It works with 1.6. We already use it for two projects. It's not officially announced. And with MapRoxy 2.0, we will add a MapRoxy daemon. It will be an optional feature. And it will not include all these features, but it will be the framework to implement all these features in the future. And summary, MapRoxy is smaller than just tile cache. There's a steady growth of features that we hope that we test them all well with all the test suites. And yeah, there's more to come in the next releases in the next years. And yeah, that's it. Thank you, Mario. Thank you.
 And I'm really happy to introduce you on the topic of the  Peace, Wonster, Sounders of Omniscape,  something that I've talked to a lot in Germany.  And yeah, I think you're presenting something like your baby  or whatever you call that.  So you're talking about past, present, and future of that talk  to you, and I'm sure they only will be  in the future.  Thank you, Till.  And hi, I'm Oliver Tonopoul.  I'm the core developer of MapRoxy,  and as Till said, I'm one of the co-founders of Omniscale  with a small company in the northwest of Germany.  And we do a lot of open source and open-sreenap stuff.  And today, I want to talk about the past, the present,  and the future of MapRoxy.  And the past.  Well, how we started with MapRoxy, it was at the end of 2008.  And we had the idea for MapRoxy.  We knew that tile caching is past,  and that it will be the future.  But we also knew that the government requires WMS services.  And our idea was we just combined a tile cache  with a full WMS and make a product out of it.  And this is how the first version of the MapRoxy  looked like as a schema.  We have, on one side, we have clients  that communicate with MapRoxy, and MapRoxy itself  communicates with other WMS servers or tile servers.  And the first features of MapRoxy, of course,  was that it supported tile requests.  So clients make requests to MapRoxy,  and MapRoxy delivers tiles.  When these tiles are missing, MapRoxy  would request the service to fill the cache.  We also supported WMS from the first version on,  cascaded request.  So MapRoxy just forwarded the request  for the same exact bounding box and WSNA  and projection to the source.  And MapRoxy would return that image.  But we also supported responding to a WMS request  from cached data.  And this is quite unique for MapRoxy.  And it does it by, maybe you can see here, just a map with  these should be tiles.  And when a request comes in, MapRoxy just combines  multiple tiles that are all affected by that request.  We'll create one image, merge the tiles together, scale  them to the exact resolution that was  requested by the user.  And it even supports the reprojection.  So when your cache is in one projection, but the user  requests another projection, it does it.  Yeah, it does the reprojection.  It does it outbound, inbound, and on the fly.  So when you have a cache in one projection, it can  reproject outbound data.  But it also can reproject on the inbound when you want to  build up a cache in one projection, but your source  doesn't support it.  So we project it then.  And on the fly for cascaded request.  And we used MapRoxy in production since 2009.  But it was only used by us.  And software only gets better when you have more users, when  you get more feedback.  And when people with other use cases come to you and say,  it would be great if MapRoxy could do that.  And so we open source MapRoxy.  And all the features are listed, where in our first  open source release.  We started with version 0.8.  And it was in May 2010.  And from then on to the present, we had now six  major releases in the last three and a half years.  And we added lots of features, most of the time to make  things easier, and also to solve completely new use cases.  And now I want to highlight some of the features that we  added in the last three and a half years.  One feature was with the 0.8.3 release that we now  supported, basically any tile source.  You can just write down your own URL template.  And you can access any XYZ tile source or WMSC that requires  a fixed parameter order.  We also improved the seeding tool.  Seeding is a process of pre-generating the tiles so  that your cache is not empty at the beginning.  And since then, the seeding tool supports parallel seeding.  So MapRoxy makes multiple retress to the same source.  And we also support a feature that we call coverages.  And with that, you only seed the required areas instead of  the complete bounding box.  And you can see it in this image.  If you want to seed the UK, and you just would use a  bounding box, you see that you're seeding a lot of water  areas and neighboring countries.  And with MapRoxy, you can just load up a polygon and just  seed within that polygon.  And this makes seeding a lot faster.  And you're saving a lot of disk space.  And you can load polygons from  geojazen files, shapefiles, even databases like  postures and Oracle, all thanks to the OGR library.  With 0.9, we added a multi-MapRoxy, which is a tool to  have multiple MapRoxy configurations run all in the  same instance of MapRoxy.  So if you want to run different WMS services, you  don't have to start up multiple MapRoxy.  Since then, we support coverage also for sources.  And I will show that later.  And we also added a demo service.  Since then, you have the ability to easily check all  your layers that you've configured with a simple  open-layers client.  With the 1.0 release, we added a security API where you can  write your own functions to define what a user is  allowed to do.  And I will also show you go into detail with that in a  minute.  And MapRoxy supports WMS feature info.  And you can optionally add XSL scripts to transform the  feature information.  So if you have two different WMS servers and the format is  different of the feature info, you can write your own  scripts to unify that format.  Then we added WMTS support and MapRoxy Utils, which is a  tool that combines multiple subcommands.  And one subcommand is the serve develop.  And with that command, you can easily start a testing  server for MapRoxy.  It will check if your MapRoxy configuration changes and  will then just reload it.  And you don't have to deal with deployment in Apache, et  cetera.  And we also support, since then, MapNIC and MapServer  directly.  So you don't have to set up a MapServer inside Apache, for  example.  With the next releases, we added a few new cache backends.  Before that, MapRoxy always stored the tiles on the file  system as simple files.  And we added MB tiles and cache to be backends.  We added more Utils, for example, the Scales tool that  allows you to convert between scales and resolutions, which  is quite handy when you deal with MapRoxy configuration.  With 1.4, one of the larger features was the clipping of  repress in the security API.  And I will show you that also in a minute.  With 1.5, we supported reprojection of tiles on the  fly.  So you have tiles in one projection, MapRoxy can  transform them into another projection.  We support mixed image mode, which is useful for aerial  images, which you want to store in SJPEC.  But at the boundaries where the aerial images are, where you  don't have aerial images, where you want to have  transparency, MapRoxy will store these tiles as PNG.  And even more MapRoxy Utils, like the Grid tool, which shows  you how many tiles are there for each zoom level in your  grid definition, and even how many tiles are within a  coverage, as you defined.  1.6 released last week, I think.  We added more cache back ends, and also the Decorate Image  API, which allows you to write your own code that can  manipulate the images to go to the client.  You could add watermarks or something like that.  So now we are at the present.  And yeah, lots of features.  We now had nearly 2,000 commits.  The change log lists 260 bullet points.  I just highlighted the most important.  And so you might ask, this must be a lot of code.  But MapRoxy itself is written in Python.  Python is a very powerful programming language.  And so the code base is relatively small.  It's just a bit over 30,000 lines of code.  And on top of that, we have nearly the same amount of  code for tests, which I think is really great for a  software like that.  We have a lot of low level tests, and we also have a lot  of high level tests, system tests that take a complete  MapRoxy configuration.  Start up a MapRoxy, make requests to it, check if the  response is valid XML document.  If everything is valid, and it also checks the request MapRoxy  makes to source request.  And we have a Jenkins instance, which runs all tests of  each commit.  It's also integrated into our GitHub repository.  So we get a warning when someone makes a pull request  that breaks the build.  And you see the small red green marker when  everything is fine.  Help, documentation now is 110G pages when you get that as  PDF.  We have a mailing list.  You can get support.  There are now a lot of local open source companies that use  MapRoxy that have knowledge in MapRoxy.  And of course, we do support for that.  So the interesting part is what can you do with it?  And now some use cases of MapRoxy.  Well, of course, it is a tile case, so you can use it for  simple tile caching.  It's used in the German broadband Atlas for one example.  The nice thing about the tile cache of MapRoxy is, of  course, the seeding tool.  You can accelerate existing WMS services.  This is used a lot in Germany, where we have a lot of old  and rusty and slow WMS service.  And they want you to accelerate them.  As I said, you can transform tiles.  And this is used by a few open street map contributors.  And they have access to open data tiles from a local tile  server.  But these tiles are not in the WebMercator projection.  And they use MapRoxy to get the data into a jostem, an  open street map editor, so that they can digitize roads  and buildings.  Another example is you have your own WMS server.  But unfortunately, that WMS doesn't support the WebMercator  projection.  But you want to overlay your WMS server with open street map  tiles, with Google Maps, et cetera.  You can use MapRoxy to re-project the data.  Now the security API.  MapRoxy can restrict access to your source services.  And it does this with a security API.  So you can write your own code that connects to your own  user database, for example.  And your code can decide what the user is allowed to do.  You can limit services, like WMS or tile request, the  request type, like feature info, and the layers, but also  the areas.  And for example, you can say, use A is allowed to see  layer X, but only within the boundaries of Germany, for  example.  And everything else is clipped.  And in this case, it's made transparent so that you can see  the layer below.  And this is quite powerful.  Satellite provider uses this to restrict the access to paid  customers.  And it also works with tile request.  And last but not least, one larger use case.  This is from an actual national network operator in  Germany.  And their customers, they have a network on the last mile  to the customers.  And that requires them to have access to official  cadastral maps.  And in case of Germany, that means Germany, we have 16  federated states.  And so we also have 16 cadastral WMS services.  So WMS is a standard, so it's quite easy to use it.  That's what you would think.  But when you look in detail to all these WMS services, you  will see that the cadastral services are mostly not free.  So you have to have some user credentials.  And they all use different authorization methods.  Some of the WMS implementations are just broken.  They don't return any G codes.  They return invalid bounding boxes.  They don't behave like the OTC standard says.  And there's not a single projection that's  handled by all, et cetera.  So when you want to make these WMS services available to  hundreds of in-house users, then good luck.  But you can use MapRoxy for that to unify all services.  You can have a single URL for all cadastral maps.  And you can use the coverage feature for all your sources.  So you can say that this source is only a  database data in this region, and when a request comes in for  this region, MapRoxy will only make a request to that  server and not to the other one.  The authorization is hidden from the user.  The capabilities are valid.  All sources are transformed to a single projection.  And we can even make the background transparent for  services that do not support that.  So make one WMS available to hundreds of in-house users.  That's easy.  OK, that's with the present and the use cases.  I now want to give you a quick look into the future.  Now 1.6 is out, so we're now working on 1.7.  We don't have any larger features planned, because we  already are making plans for the 2.0 release.  And we will make some backwards incompatible changes  used to refactor some parts.  We will also support Python 3.  And we will add a new background service.  And we will support Python 3.  We will skip the 2.5.  And new background service.  This is the last point I want to make.  First, I have to explain the process model of MapRoxy.  MapRoxy is a retrace base.  The retrace comes in when a threat deals with a  process and the response goes out.  But as soon as the response is out, you can't do anything  anymore, because the next request comes in.  And this is given by the web service gateway interface  that MapRoxy implements.  And it's similar to how CGI and Apache modules and  Java setlets work.  So limitations are that you don't have any long-running  processes.  As soon as the request ends, your process stops.  And you can't really communicate between requests.  There are workarounds.  MapRoxy uses, for example, an external tool to have a  long-running process to see tiles.  And MapRoxy also uses file logs for some synchronization,  so that not two processes make the same  request to the source.  But it's all limited.  If we had a long-running background service, we could  implement all these features.  We could add something like a priority queue, so that we  could limit the load to the sources.  Now, because one process would now, OK, we have two  running queries to the source.  We are not making any more.  We're just queuing that.  We could add seeding without having cron task, et cetera.  We could rate limit request and implement adaptive  pre-seeding, et cetera.  So a lot of features that would be possible with that.  And we have a MapRoxy render daemon that's available at  GitHub.  It works with 1.6.  We already use it for two projects.  It's not officially announced.  And with MapRoxy 2.0, we will add a MapRoxy daemon.  It will be an optional feature.  And it will not include all these features, but it will  be the framework to implement all these features in the  future.  And summary, MapRoxy is smaller than just tile cache.  There's a steady growth of features that we hope that  we test them all well with all the test suites.  And yeah, there's more to come in the next releases in the  next years.  And yeah, that's it.  Thank you, Mario.  Thank you.
MapProxy_--_Past__Present__Future__720p__0_1343.png MapProxy_--_Past__Present__Future__720p__1344_5651.png MapProxy_--_Past__Present__Future__720p__5652_7031.png MapProxy_--_Past__Present__Future__720p__7032_11039.png MapProxy_--_Past__Present__Future__720p__11040_13679.png MapProxy_--_Past__Present__Future__720p__13680_15335.png MapProxy_--_Past__Present__Future__720p__15336_15515.png MapProxy_--_Past__Present__Future__720p__15516_20255.png MapProxy_--_Past__Present__Future__720p__20256_20759.png MapProxy_--_Past__Present__Future__720p__20760_25667.png MapProxy_--_Past__Present__Future__720p__25668_27335.png MapProxy_--_Past__Present__Future__720p__27336_28739.png MapProxy_--_Past__Present__Future__720p__28740_29267.png MapProxy_--_Past__Present__Future__720p__29268_30119.png MapProxy_--_Past__Present__Future__720p__30120_33040.png