How to GIS with Python
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Lehmann, Anders
Anders Lehmann - How to GIS with Python In this talk I will present some tools for working with Geographic Information Systems in Python. Geographic information Systems are widely used for managing geographic (map) data. As an example I will present how to use Open Street Map data, in routing, traffic planning and estimation of pollution emission. For the purpose of the project EcoSense, GPS data from users smartphones are mapped to OSM roads. The map matching algorithm is written in Python and uses data from the database PostgreSQL, with the PostGIS extension. One of the goals of the EcoSense project is to devise methods to improve the estimation of air quality in urban environments.
 Good morning. Thank you. So, I call this how to GIS in Python with a subtitle of A Tale of Two Cities. It was supposed to be about the city of my university, Ahus and Istanbul, but due to time range I have chosen only to talk about Istanbul. It's the more interesting case anyways. So, the subject is a little bit wrong, but it was kind of funny. Okay, let's go on. So, about me, my name is Anas Lieman. This is a compact edition of my CV. Well, it's not very important, but now I'm working at Ahus University in the Engineering School of Ahus, and I'm teaching electronic engineers physics and programming. So, and part of my work is to finish my PhD. So, in one year's time I have to deliver my dissertation, and I'm doing the PhD in the context of a funded project called Ecosense, which is about collective mobile sensing and modeling of emissions from mainly traffic. So, and the emissions is, there's two kinds, two kinds of emissions you are, we are interested in. The one is the climate gases, the CO2 and the methane and stuff like that, and the other kind of emissions we are interested in are the more pollutant, the more, more acutely toxic pollution coming from car traffic. There's also a visualization part of that project, which I'm not a part of. Okay, but the idea is that you ask people to use a special app on the mobile phones, and then we get data about how they are moving around in the cities, in the country, and we can use these data to make models. So, the contents of this talk is like this, so a little short introduction to what GIS is. I'm not going to go into all these, there's a lot of detail in GIS, and I'm going to just hand wave on that. I'll look at a couple of applications for GIS. I think I'm more interested in applications, so that's my forte, you could say. A little bit about where you can find data to use on your GIS systems or applications, something about the patent tools available, some examples, and then I will go and if I have time to talk about my research on building a transportation model for Istanbul. Okay, so, yeah, let's get on with it. So, what is GIS? Well, GIS stands for geographical information systems, and it's all about maps. There you go, that is simple. Well, not just a moment, because we live on a sphere, our Earth is spherical, and we like our maps to be flat, two-dimensional. So, how do you fit a sphere on a piece of paper? That's where all the nitty gritty things are coming from in GIS, how to project the spherical things on our Earth into a two-dimensional map. And it turns out that there are a lot of different ways to do these protections, and each of these ways have certain properties which are good properties, but there are no projections which can have all the good properties that you can do. So, some projections retain area, some retain length and stuff like that. So depending on your application, you will need different kind of projections. But, luckily, this has mainly been solved in my view. There's a database containing 4,000 different projections, and there is a standard way to convert from one map type to another map type. So, if you have legacy maps, you just need to know which kind of projection that map is using, and then you can turn the data into the map projection that you need. The most, well, most modern maps are using this WST84 projection. And so, if you start a new project, then it might be a good idea to try to use this projection, which is kind of standard. So, the nitty gritty part of GIS is mainly this projection part. There's a lot of mathematics and stuff involved in that, and it's kind of hard and confusing. So, I'm just going to punt that to someone who knows about it and go on with the applications. So, there are quite a lot of different applications for GIS. This is just one application. This is from Denmark. All the black dots here are the street lamps in my area. So, someone, the municipality has to keep track of their inventory of street lamps. They have made a map where they have put in all the locations of the street lamps. So, that's a way of keeping track of your assets, and it's a way that they can use for planning. They know how many street lamps they have and then know where they are. And if they have to be serviced or something like that, you can plan in how many cars you need to service these lamps and in which order would it be a nice thing to do. So, that's one application. The assets track of things that you put out in your environment. Another way, another asset tracking application is to keep track of where you put down your sewage lines, your underground cable, your electric cables and stuff like that, so that you hopefully can educate contractors so that they're not ruining anything when they have to dig into the ground. Fleet management, this is more a dynamic tracking thing, but you still, you would like to know where all your taxes or cars or trucks, where they are and where they are going. So, you need to put them on the map, maybe dynamically, but still on a map. And if you're a city planner, you would like to have accurate maps so that you can plan for different zones in the city. Where should the industrial facilities be and where should the residential area be and stuff like that. And in these kinds of maps, you also need features of the geography. Is this hilly? Are there swamps? And what kind of, what is the ground made of on these specific sites? So, that's another way of putting information into maps that you can use for GIS things. Okay. And of course, routing, which I think is very important and very used nowadays, very used application of GIS. We do it on our smartphones, we do it on specialized GPS route finders in our cars and stuff like that. So, it has been a very successful application of GIS. And of course, we call it GPS receivers because, but the GPS only give us a position. We need the maps to actually find a way from one point to another and preferably the fastest one. I already mentioned city planning, but there is a specialized version of city planning, which is traffic planning, traffic modeling. This is where I have done some research in, and the problem is of course that we want to plan our city so that we get as little congestion as possible in order to plan for congestion. We need to have a model for where is the traffic going and when is it going there. So, all right. Where can we find data for our maps? Well, there is a very important resource called OpenStreetMap. I don't know if you know it, but I think that it might be undervalued and underused because Google Earth is of course very, Google Maps is very good. But the OpenStreetMap has this further, it's an open format and you can actually go in and change stuff if you want. If you live in an area and you build a new road or the closer road, you can actually go in and change that in your map, in the OpenStreetMap and it's very easy to do that. So, for researchers I think that's a very important thing that the data is available and it's open and free to use. Of course, I can use Google Maps for just for static maps, but I can't use it for getting the features of the map. So, I can see a map where there are roads, but I can't really get the coordinates of these roads. It's a little bit harder to do that actually. But there are other sources, at least in Denmark we have national data centers for a lot of different maps. There's a center called Environment GIS which has all kinds of strange information about where the rivers are and where the where they are deposed for toxic waste and stuff like that. So, you can go there if you want to research that. The municipalities also have these sources and that can actually be sometimes a problem because they tend to be quite old, some of these databases and they there you can actually risk to have a non-standard projection mode and you can, it can be quite hard to convert these legacy databases to a more modern projection so that you can use OpenStreetMap for instance with the old data from the legacy database. So, I just wanted to mention, I found this database the other day about where all the cell towers for mobile phones are in Denmark. There's a database so you could find the nearest cell tower if you knew where you were. I think there must be applications for that also. Okay, so this is not easy to see but the patent tools that I have used the most is this, it's called Q, quantum GIS, this is a visualization program with an embedded Python interpreter. So you can make all your scripts in Python. I'm sorry about the colors here, I thought that, yeah, but maybe we could just make a little demo here. So, now it's the same image just build a green color instead of the light blue. This is actually all the roads in Istanbul. So there are 300,000 roads in Istanbul and that's the only thing that's covered here. So you can actually see the Pospers Street where there are no roads and there are some islands here with some roads on. And then there are some mountains where there are no roads. But this visualization program is actually fast enough to accommodate this quite large number of roads so you can see it redraws quite fast. So the other tool I would mention is the Arc GIS, it's a commercial product which also has an embedded Python interpreter. This is, it seems very popular in research. I've read several papers where they're using Arc GIS. But I think that I found QGIS a bit more approachable from the Python side so I chose to introduce that. So, yeah. So I'm going to talk a little bit more about tools. These are not Python tools per se, but they are very good bindings to them so you can accept them through Python so that they are not really Python. The Postgres database with the extensions called PostGIS and PG routing are quite good. So the Postgres is of course a very good SQL database and the PostGIS extension gives a lot of relevant functions to manipulate GIS data. So I'm not going into the details but because that's a whole new talk I think. And on top of PostGIS this PG routing extension has been built so that you can, if you have, if you put your data into the database you can start to find the shortest paths from one point to another using this extension. In order to do all the GIS conversion from, there are a lot of different formats. Not only projections but also data formats. But there are these libraries available, DIL and especially OGR to OGR which can reach almost any relevant format and convert to most of the other relevant formats. So they are very generic tools but they are not that hard to use actually. And there are Python bindings for both of them. Okay, so I talked about OpenStreetMap and I'm just going to show you hopefully. So I should, there you go. So when you OpenStreetMap you get maps of course and I've zoomed into BitVal. Maybe you can recall or recognize the river and venue. What is quite hard to see with these colors is that there's actually some problems with this map because there is a, you can walk like along here and actually you can go under the bridge and you can continue on this footpath here. But there is no connection from this bridge to this footpath. But that's not hard to change. You can just press edit and if you have logged in and stuff like that, you are presented with this view and you can then choose to connect this footpath here. Oh, sorry. I have to press with this footpath. And then you just have to do that once more. So now I have put in a new line here and I should put in the metadata as well that this is a, this is, it's a path I guess. So when I have put in, I don't know if it has a name or if I should prepare the surface and stuff like that. But there are a lot of metadata that you can put in to OpenStreetMap and when you're satisfied you can just press save and then it will be updated to the database. So and I want you to recognize that it is very easy and it's actually could be beneficial for the, for, for at least for me as a researcher if the OpenStreetMap is as correct as possible. So I want to urge you to, when you've noticed something changed in your local environment, please try to, to go into the OpenStreetMap and we can take it later, right? And change it. It's very easy and it's very nice when it's working, when it's correct. So let's go back to the presentation. So now we come to my, to my stuff, the traffic model that I've been working with and yeah, of course this might be very boring for you because this is of course my pet. So maybe I go into too much detail about this. But the, the reason why we want to have a traffic model is in, we want to be able to predict how the traffic flows through a network. We want to be able to predict where the congestions are and, and we, especially, we want to predict how, if we change the network, how would that, how would that change the flow through the network. So what I've been working with is a model that, that considers how people actually choose where to drive. So it's kind of a selfish margin model for, for drivers. And in, in the research, it's usually called the assignment problem. It's how to assign drivers to different routes in a network. The, the basic principles are derived from econometrics and it's based on, on finding illiter, equilibrium values, and steady state solutions. So it's, it's kind of, it's kind of a static model. So you, you, you create, you, you find out what is the demand for, for going from one place to another. And then you try to find the steady state solution for, for that network. So it's not dynamic per se. But it's the, the, the big problem with, with these kind of models is that there are actually a very large number of different possible routes. And if you want to make a computer, find the, the, how people are distributed through all these routes is actually a, a large problem. So this is a, this is a picture of Copenhagen, or the central part of Copenhagen anyway. So at the bottom right, we have the Copenhagen airport and there is a motorway coming down from the north. So this is an assignment to, to see how will the, how will traffic flow through the Copenhagen area in, in the morning when people come from the north and I want, they want to go to the, to the airport. So the large path here is the motorway. There's a motorway all the way around Copenhagen and it goes to the airport. So of course most people choose to, to go about it like that. But as, as many as more and more people come onto the motorway, it, it turns out that the, the traffic flows and then it's actually, so there is other ways that are quite as fast. And they are going to through the center of, of the Copenhagen, which might not be that nice for, for the, for the morning, morning traffic in the central Copenhagen. So, but the idea here is that we try to find out how many are going at each route. And when we have an equilibrium, everyone would have the same travel time. That's the basic idea because then we have what we call a user equilibrium. Okay. So how, how do we model congestion? Well, we, we know that, that roads have a capacity. They are designed to, to have a certain amount of cars per hour. This is the C A in the bottom here. And when we reach the capacity, we know that the traffic flows and we model that with this simple power formula. So the time it takes to go through a link is given by the free flow time. That's the flow we would normally, we would experience if there were no other cars on the road. And then we have this power function where when, when the volume of the cars are approaching the capacity, then we, it takes long, long, long time. So this is in a very old formula, but it has worked for, it still works. And it's 50 years old or something like that. It gives actually a very good approximation on how traffic flows. And we can use this idea if we, now we have a two routes between our origin and destination. And we want to assign traffic to these two routes in such a way that the travel time is the same on these two routes. And we can find the equilibrium point where these two congestion curves, across. That's the basic idea. The user equilibrium is the assignment where everyone has the same travel time. Because if we all have some travel time, we can't, we can't have a better travel time by choosing another route. That's the basic idea. It's, I think it's attributable to Nash equilibrium, if you heard about them. So it ties into all this game theory and economic physics and selfish users and stuff like that. User equilibrium, the different term, inistic part here is only applicable in, in very small networks because you need to consider all, all possible routes. And as in even small networks, there's a lot of different possible routes. So it's, it's not really feasible to do deterministic user models. So people have invented stochastic methods where you can look at, well, the basic formulation is that instead of being certain that you have the best possible route, the shortest time, we can, we can put in some stochasticity and then we can, then people think they have the best route. So, but there's a bit of uncertainty and this uncertainty we can model and we can, we can use that to, to drive the assignment process. So it turns into, since we are looking for the smallest travel time, it's extreme value problem in stochastic. And we can, there are different formulation for it, but the one I've worked with is this path size loaded where you, where you look at how, how path are looking, if they are looking like if different routes use the same links. So, so the stochastic user equilibrium part is nice because you can use stochastic methods and, but there are still, the problem with the stochastic user equilibrium is that there will be a non-zero probability to use every route. So it might be a very small probability that you use a very stupid route, but it's, it will be there and it will be calculated. So you, you, you kind of underestimate the good parts, the good routes. So that's not so nice. So in the Istanbul case, I have used data from, from, from Open Street Map. So you can just, there's, there's a way that you can actually, you can, you don't need to have the complete, the complete data set. You can just pick what area that you want the data from and then it's, it's converted into a poly, so that you have a network where everything is connected and the, and it's turned into segments instead of roads. So yeah, this is, it's a bit of detail. But there are 300,000 road segments. I have done a simulation with 2000 origin destination pair. I used the Postgres thing and have implemented a combination of, of the deterministic user equilibrium and the stochastic user equilibrium so that we, so that I can assign traffic to different routes. And I used QGIS for the visualization. So, and I think it's kind of nice. You can see on the left-hand side here that you can see the elephant. That means that there's a direct connection from the QGIS to the Postgres database. And I can just point QGIS to my database and then ask it to show all the routes in my solution set. And this is from, this is how I have, how my algorithm has assigned the traffic to, to the different routes. So, of course it's, yeah. There's a lot of detail that I can't really, I haven't time to, to cover here. So, that was my demo. Okay. So, in conclusion, so the way that I have used Python is to drive all the other tools. There are, there are libraries for doing routing directly in Python. But it was actually very easy and fast to use Postgres. So, I just, I just did that. So, there are many tools. And now I'm prompted to end my talk. Yeah. It's confusing, but it's doable. I think that's my main conclusion. So, thank you. Okay. Do you have any, do you have any questions? Comments? Thank you for the talk. You, you do a lot of theoretical research and traffic congestion, but do you actually get signals from drivers? And do, and my question is mainly because companies get that from their devices like Nokia or Google, they get that from, from users. But you as an independent researcher, where do you get the signals from? Well, I'm part of this Ecosense project. And so, there were, in this project, there were three PhD students. One should do the, the mobile applications. And one did the modeling, that's me. And one did the visualization. So, we have, we have made this library for, so it's easy to make applications that send us data. And we have made some applications for, to get users to use it. So, there's an application called are you e ready, where you can download this application, use it for a month. And then there will be an analysis if your traffic pattern is fit for a e-car. And if it is, you can borrow an e-car for a month back after that. So, that have given us some data. There's another project called the, Herning drives to the moon, or bicycles to the moon, where we, but the municipality of Herning, it's a small town where I teach. They want people to bicycle to, to work. And then they have, they ask people to use this application, which will measure the, the length of the bike distance, and then accumulate it to hopefully get it to, to, so that they could bike to the moon and back. That also gives us the data, also when they are driving in their cars. So, so there, there have been several small scale applications like that, that gives us some data, which we, which I can use to, to extract routes and see where people are actually moving. And I, I get the speed information as well. So, I can also guess where there is congestion. So, so I can, I can use this data to also see if my, my models are correct. Okay. More questions? There's one here during the talk. I have one question, one comment, a question. Okay. I use Postgrease for large lists of, of points, geographical points and areas and lines. Is there something directly in Python that, what would you recommend if I have a few thousands of points and I want to find the closest ones or to group them? What library would you recommend? To store longitude, latitude and so on? Yeah. Well, I look briefly at the PyRoute, which is a, has a routing facility, I guess. I'm not, I don't remember the, the data format that they use for storage. But there are, you could look for Dijkstra. There would be several implementations of, of, of Dijkstra, Dijkstra path finding, route finding algorithm. So you could use that, of course. Okay. Thank you. And a comment to OpenStreetMap. Yes. It's actually not a map, it's a database of geographical data. Yes. Fully available. You have just now, you did an update to this path here in Bilbao. In between the increasing number of contributors to OpenStreetMap and the increased complexity of all the routes you can do, you can make a lot of further attributes is some, sometimes it's too much for new users and not inviting them because they, oh, I have a lot of things to learn. In OpenStreetMap, there is a nice feature where you even don't have to, to sign up. Mark a problem or a bug, submit a bug. It is directly on the page and you can just put a marker anywhere on the map and write in your language. Yeah. There is some change to it. This is false and there is always someone just like you or me or anyone else who goes to the area and then looks at this and fixes the problem on the best way possible. Yeah. I had to do it for the Istanbul case because I noticed that the routes that my algorithm found was, well, it was not how the taxi drivers and the bus drivers drove. So I had to find out why and then it turned out that there was a missing roundabout. So I just put that in. So, yeah. There's also another feature which I think is a good feature for OpenStreetMap. They are trying to automatically find problems. So for instance, if a footpath crosses a motorway, that should not happen. So if they mark it with an arrow, so there are a lot of automatic features to find problems. So if you are bored, you can also look at all these in your own area. You can look for the errors in OpenStreet. There is a special map where all the errors is pointed out. So you could just correct them if you are native to that area. Okay. Okay. That would be the time to move to the next talk and let's thank this speaker again. Thank you. Next talk will be in four minutes.
 Good morning. Thank you. So, I call this how to GIS in  Python with a subtitle of A Tale of Two Cities. It was supposed to be about the city of my  university, Ahus and Istanbul, but due to time range I have chosen only to talk about  Istanbul. It's the more interesting case anyways. So, the subject is a little bit wrong, but  it was kind of funny. Okay, let's go on. So, about me, my name is Anas Lieman. This is  a compact edition of my CV. Well, it's not very important, but now I'm working at Ahus  University in the Engineering School of Ahus, and I'm teaching electronic engineers physics  and programming. So, and part of my work is to finish my PhD. So, in one year's time I  have to deliver my dissertation, and I'm doing the PhD in the context of a funded project  called Ecosense, which is about collective mobile sensing and modeling of emissions from  mainly traffic. So, and the emissions is, there's two kinds, two kinds of emissions you are,  we are interested in. The one is the climate gases, the CO2 and the methane and stuff like  that, and the other kind of emissions we are interested in are the more pollutant, the  more, more acutely toxic pollution coming from car traffic. There's also a visualization  part of that project, which I'm not a part of. Okay, but the idea is that you ask people  to use a special app on the mobile phones, and then we get data about how they are moving  around in the cities, in the country, and we can use these data to make models. So, the  contents of this talk is like this, so a little short introduction to what GIS is. I'm not  going to go into all these, there's a lot of detail in GIS, and I'm going to just hand  wave on that. I'll look at a couple of applications for GIS. I think I'm more interested in applications,  so that's my forte, you could say. A little bit about where you can find data to use on  your GIS systems or applications, something about the patent tools available, some examples,  and then I will go and if I have time to talk about my research on building a transportation  model for Istanbul. Okay, so, yeah, let's get on with it. So, what is GIS? Well, GIS  stands for geographical information systems, and it's all about maps. There you go, that  is simple. Well, not just a moment, because we live on a sphere, our Earth is spherical,  and we like our maps to be flat, two-dimensional. So, how do you fit a sphere on a piece of  paper? That's where all the nitty gritty things are coming from in GIS, how to project  the spherical things on our Earth into a two-dimensional map. And it turns out that there are a lot  of different ways to do these protections, and each of these ways have certain properties  which are good properties, but there are no projections which can have all the good properties  that you can do. So, some projections retain area, some retain length and stuff like that.  So depending on your application, you will need different kind of projections. But, luckily,  this has mainly been solved in my view. There's a database containing 4,000 different projections,  and there is a standard way to convert from one map type to another map type. So, if you  have legacy maps, you just need to know which kind of projection that map is using, and  then you can turn the data into the map projection that you need. The most, well, most modern  maps are using this WST84 projection. And so, if you start a new project, then it might  be a good idea to try to use this projection, which is kind of standard. So, the nitty gritty  part of GIS is mainly this projection part. There's a lot of mathematics and stuff involved  in that, and it's kind of hard and confusing. So, I'm just going to punt that to someone  who knows about it and go on with the applications. So, there are quite a lot of different applications  for GIS. This is just one application. This is from Denmark. All the black dots here are  the street lamps in my area. So, someone, the municipality has to keep track of their  inventory of street lamps. They have made a map where they have put in all the locations  of the street lamps. So, that's a way of keeping track of your assets, and it's a way that  they can use for planning. They know how many street lamps they have and then know where  they are. And if they have to be serviced or something like that, you can plan in how  many cars you need to service these lamps and in which order would it be a nice thing  to do. So, that's one application. The assets track of things that you put out in your environment.  Another way, another asset tracking application is to keep track of where you put down your  sewage lines, your underground cable, your electric cables and stuff like that, so that  you hopefully can educate contractors so that they're not ruining anything when they have  to dig into the ground. Fleet management, this is more a dynamic tracking thing, but  you still, you would like to know where all your taxes or cars or trucks, where they are  and where they are going. So, you need to put them on the map, maybe dynamically, but  still on a map. And if you're a city planner, you would like to have accurate maps so that  you can plan for different zones in the city. Where should the industrial facilities be  and where should the residential area be and stuff like that. And in these kinds of maps,  you also need features of the geography. Is this hilly? Are there swamps? And what kind  of, what is the ground made of on these specific sites? So, that's another way of putting information  into maps that you can use for GIS things. Okay. And of course, routing, which I think  is very important and very used nowadays, very used application of GIS. We do it on  our smartphones, we do it on specialized GPS route finders in our cars and stuff like that.  So, it has been a very successful application of GIS. And of course, we call it GPS receivers  because, but the GPS only give us a position. We need the maps to actually find a way from  one point to another and preferably the fastest one. I already mentioned city planning, but  there is a specialized version of city planning, which is traffic planning, traffic modeling.  This is where I have done some research in, and the problem is of course that we want  to plan our city so that we get as little congestion as possible in order to plan for  congestion. We need to have a model for where is the traffic going and when is it going  there. So, all right. Where can we find data for our maps? Well, there is a very important  resource called OpenStreetMap. I don't know if you know it, but I think that it might  be undervalued and underused because Google Earth is of course very, Google Maps is very  good. But the OpenStreetMap has this further, it's an open format and you can actually go  in and change stuff if you want. If you live in an area and you build a new road or the  closer road, you can actually go in and change that in your map, in the OpenStreetMap and  it's very easy to do that. So, for researchers I think that's a very important thing that  the data is available and it's open and free to use. Of course, I can use Google Maps for  just for static maps, but I can't use it for getting the features of the map. So, I can  see a map where there are roads, but I can't really get the coordinates of these roads.  It's a little bit harder to do that actually. But there are other sources, at least in Denmark  we have national data centers for a lot of different maps. There's a center called Environment  GIS which has all kinds of strange information about where the rivers are and where the  where they are deposed for toxic waste and stuff like that. So, you can go there if you  want to research that. The municipalities also have these sources and that can actually  be sometimes a problem because they tend to be quite old, some of these databases and  they there you can actually risk to have a non-standard projection mode and you can,  it can be quite hard to convert these legacy databases to a more modern projection so that  you can use OpenStreetMap for instance with the old data from the legacy database. So,  I just wanted to mention, I found this database the other day about where all the cell towers  for mobile phones are in Denmark. There's a database so you could find the nearest cell  tower if you knew where you were. I think there must be applications for that also.  Okay, so this is not easy to see but the patent tools that I have used the most is this, it's  called Q, quantum GIS, this is a visualization program with an embedded Python interpreter.  So you can make all your scripts in Python. I'm sorry about the colors here, I thought  that, yeah, but maybe we could just make a little demo here. So, now it's the same image  just build a green color instead of the light blue. This is actually all the roads in Istanbul.  So there are 300,000 roads in Istanbul and that's the only thing that's covered here.  So you can actually see the Pospers Street where there are no roads and there are some  islands here with some roads on. And then there are some mountains where there are no roads.  But this visualization program is actually fast enough to accommodate this quite large  number of roads so you can see it redraws quite fast. So the other tool I would mention is  the Arc GIS, it's a commercial product which also has an embedded Python interpreter. This  is, it seems very popular in research. I've read several papers where they're using Arc  GIS. But I think that I found QGIS a bit more approachable from the Python side so I chose  to introduce that. So, yeah. So I'm going to talk a little bit more about tools. These  are not Python tools per se, but they are very good bindings to them so you can accept  them through Python so that they are not really Python. The Postgres database with the extensions  called PostGIS and PG routing are quite good. So the Postgres is of course a very good SQL  database and the PostGIS extension gives a lot of relevant functions to manipulate GIS  data. So I'm not going into the details but because that's a whole new talk I think. And  on top of PostGIS this PG routing extension has been built so that you can, if you have,  if you put your data into the database you can start to find the shortest paths from  one point to another using this extension. In order to do all the GIS conversion from,  there are a lot of different formats. Not only projections but also data formats. But  there are these libraries available, DIL and especially OGR to OGR which can reach almost  any relevant format and convert to most of the other relevant formats. So they are very  generic tools but they are not that hard to use actually. And there are Python bindings  for both of them. Okay, so I talked about OpenStreetMap and I'm just going to show you  hopefully. So I should, there you go. So when you OpenStreetMap you get maps of course  and I've zoomed into BitVal. Maybe you can recall or recognize the river and venue. What  is quite hard to see with these colors is that there's actually some problems with this  map because there is a, you can walk like along here and actually you can go under the  bridge and you can continue on this footpath here. But there is no connection from this  bridge to this footpath. But that's not hard to change. You can just press edit and if  you have logged in and stuff like that, you are presented with this view and you can then  choose to connect this footpath here. Oh, sorry. I have to press with this footpath.  And then you just have to do that once more. So now I have put in a new line here and I  should put in the metadata as well that this is a, this is, it's a path I guess. So when  I have put in, I don't know if it has a name or if I should prepare the surface and stuff  like that. But there are a lot of metadata that you can put in to OpenStreetMap and when  you're satisfied you can just press save and then it will be updated to the database.  So and I want you to recognize that it is very easy and it's actually could be beneficial  for the, for, for at least for me as a researcher if the OpenStreetMap is as correct as possible.  So I want to urge you to, when you've noticed something changed in your local environment,  please try to, to go into the OpenStreetMap and we can take it later, right? And change  it. It's very easy and it's very nice when it's working, when it's correct. So let's  go back to the presentation. So now we come to my, to my stuff, the traffic model that  I've been working with and yeah, of course this might be very boring for you because  this is of course my pet. So maybe I go into too much detail about this. But the, the reason  why we want to have a traffic model is in, we want to be able to predict how the traffic  flows through a network. We want to be able to predict where the congestions are and,  and we, especially, we want to predict how, if we change the network, how would that,  how would that change the flow through the network. So what I've been working with is  a model that, that considers how people actually choose where to drive. So it's kind of a  selfish margin model for, for drivers. And in, in the research, it's usually called the  assignment problem. It's how to assign drivers to different routes in a network. The, the  basic principles are derived from econometrics and it's based on, on finding illiter, equilibrium  values, and steady state solutions. So it's, it's kind of, it's kind of a static model.  So you, you, you create, you, you find out what is the demand for, for going from one  place to another. And then you try to find the steady state solution for, for that network.  So it's not dynamic per se. But it's the, the, the big problem with, with these kind  of models is that there are actually a very large number of different possible routes.  And if you want to make a computer, find the, the, how people are distributed through all  these routes is actually a, a large problem. So this is a, this is a picture of Copenhagen,  or the central part of Copenhagen anyway. So at the bottom right, we have the Copenhagen  airport and there is a motorway coming down from the north. So this is an assignment to,  to see how will the, how will traffic flow through the Copenhagen area in, in the morning  when people come from the north and I want, they want to go to the, to the airport. So  the large path here is the motorway. There's a motorway all the way around Copenhagen and  it goes to the airport. So of course most people choose to, to go about it like that.  But as, as many as more and more people come onto the motorway, it, it turns out that the,  the traffic flows and then it's actually, so there is other ways that are quite as fast.  And they are going to through the center of, of the Copenhagen, which might not be that  nice for, for the, for the morning, morning traffic in the central Copenhagen. So, but  the idea here is that we try to find out how many are going at each route. And when we  have an equilibrium, everyone would have the same travel time. That's the basic idea because  then we have what we call a user equilibrium. Okay. So how, how do we model congestion?  Well, we, we know that, that roads have a capacity. They are designed to, to have a  certain amount of cars per hour. This is the C A in the bottom here. And when we reach the  capacity, we know that the traffic flows and we model that with this simple power formula.  So the time it takes to go through a link is given by the free flow time. That's the  flow we would normally, we would experience if there were no other cars on the road. And  then we have this power function where when, when the volume of the cars are approaching  the capacity, then we, it takes long, long, long time. So this is in a very old formula,  but it has worked for, it still works. And it's 50 years old or something like that.  It gives actually a very good approximation on how traffic flows. And we can use this  idea if we, now we have a two routes between our origin and destination. And we want to  assign traffic to these two routes in such a way that the travel time is the same on  these two routes. And we can find the equilibrium point where these two congestion curves,  across. That's the basic idea. The user equilibrium is the assignment where everyone has the same  travel time. Because if we all have some travel time, we can't, we can't have a better travel  time by choosing another route. That's the basic idea. It's, I think it's attributable  to Nash equilibrium, if you heard about them. So it ties into all this game theory and economic  physics and selfish users and stuff like that. User equilibrium, the different term, inistic  part here is only applicable in, in very small networks because you need to consider all,  all possible routes. And as in even small networks, there's a lot of different possible  routes. So it's, it's not really feasible to do deterministic user models. So people  have invented stochastic methods where you can look at, well, the basic formulation is  that instead of being certain that you have the best possible route, the shortest time,  we can, we can put in some stochasticity and then we can, then people think they have the  best route. So, but there's a bit of uncertainty and this uncertainty we can model and we can,  we can use that to, to drive the assignment process. So it turns into, since we are looking  for the smallest travel time, it's extreme value problem in stochastic. And we can, there  are different formulation for it, but the one I've worked with is this path size loaded  where you, where you look at how, how path are looking, if they are looking like if different  routes use the same links. So, so the stochastic user equilibrium part is nice because you  can use stochastic methods and, but there are still, the problem with the stochastic  user equilibrium is that there will be a non-zero probability to use every route. So it might  be a very small probability that you use a very stupid route, but it's, it will be there  and it will be calculated. So you, you, you kind of underestimate the good parts, the good  routes. So that's not so nice. So in the Istanbul case, I have used data from, from, from Open  Street Map. So you can just, there's, there's a way that you can actually, you can, you  don't need to have the complete, the complete data set. You can just pick what area that  you want the data from and then it's, it's converted into a poly, so that you have a  network where everything is connected and the, and it's turned into segments instead  of roads. So yeah, this is, it's a bit of detail. But there are 300,000 road segments.  I have done a simulation with 2000 origin destination pair. I used the Postgres thing  and have implemented a combination of, of the deterministic user equilibrium and the  stochastic user equilibrium so that we, so that I can assign traffic to different routes.  And I used QGIS for the visualization. So, and I think it's kind of nice. You can see  on the left-hand side here that you can see the elephant. That means that there's a direct  connection from the QGIS to the Postgres database. And I can just point QGIS to my database  and then ask it to show all the routes in my solution set. And this is from, this is  how I have, how my algorithm has assigned the traffic to, to the different routes. So,  of course it's, yeah. There's a lot of detail that I can't really, I haven't time to, to  cover here. So, that was my demo. Okay. So, in conclusion, so the way that I have used  Python is to drive all the other tools. There are, there are libraries for doing routing  directly in Python. But it was actually very easy and fast to use Postgres. So, I just,  I just did that. So, there are many tools. And now I'm prompted to end my talk. Yeah.  It's confusing, but it's doable. I think that's my main conclusion. So, thank you.  Okay. Do you have any, do you have any questions? Comments?  Thank you for the talk. You, you do a lot of theoretical research and traffic congestion,  but do you actually get signals from drivers? And do, and my question is mainly because  companies get that from their devices like Nokia or Google, they get that from, from  users. But you as an independent researcher, where do you get the signals from?  Well, I'm part of this Ecosense project. And so, there were, in this project, there  were three PhD students. One should do the, the mobile applications. And one did the modeling,  that's me. And one did the visualization. So, we have, we have made this library for,  so it's easy to make applications that send us data. And we have made some applications  for, to get users to use it. So, there's an application called are you e ready, where  you can download this application, use it for a month. And then there will be an analysis  if your traffic pattern is fit for a e-car. And if it is, you can borrow an e-car for  a month back after that. So, that have given us some data. There's another project called  the, Herning drives to the moon, or bicycles to the moon, where we, but the municipality  of Herning, it's a small town where I teach. They want people to bicycle to, to work. And  then they have, they ask people to use this application, which will measure the, the length  of the bike distance, and then accumulate it to hopefully get it to, to, so that they  could bike to the moon and back. That also gives us the data, also when they are driving  in their cars. So, so there, there have been several small scale applications like that,  that gives us some data, which we, which I can use to, to extract routes and see where  people are actually moving. And I, I get the speed information as well. So, I can also  guess where there is congestion. So, so I can, I can use this data to also see if my, my  models are correct.  Okay. More questions? There's one here during the talk.  I have one question, one comment, a question. Okay. I use Postgrease for large lists of,  of points, geographical points and areas and lines. Is there something directly in Python  that, what would you recommend if I have a few thousands of points and I want to find  the closest ones or to group them? What library would you recommend? To store longitude, latitude  and so on?  Yeah. Well, I look briefly at the PyRoute, which is a, has a routing facility, I guess.  I'm not, I don't remember the, the data format that they use for storage. But there are,  you could look for Dijkstra. There would be several implementations of, of, of Dijkstra,  Dijkstra path finding, route finding algorithm. So you could use that, of course.  Okay. Thank you. And a comment to OpenStreetMap.  Yes.  It's actually not a map, it's a database of geographical data.  Yes.  Fully available. You have just now, you did an update to this path here in Bilbao. In  between the increasing number of contributors to OpenStreetMap and the increased complexity  of all the routes you can do, you can make a lot of further attributes is some, sometimes  it's too much for new users and not inviting them because they, oh, I have a lot of things  to learn. In OpenStreetMap, there is a nice feature where you even don't have to, to sign  up. Mark a problem or a bug, submit a bug. It is directly on the page and you can just  put a marker anywhere on the map and write in your language. Yeah. There is some change  to it. This is false and there is always someone just like you or me or anyone else  who goes to the area and then looks at this and fixes the problem on the best way possible.  Yeah. I had to do it for the Istanbul case because I noticed that the routes that my  algorithm found was, well, it was not how the taxi drivers and the bus drivers drove.  So I had to find out why and then it turned out that there was a missing roundabout. So  I just put that in. So, yeah. There's also another feature which I think is a good feature  for OpenStreetMap. They are trying to automatically find problems. So for instance, if a footpath  crosses a motorway, that should not happen. So if they mark it with an arrow, so there  are a lot of automatic features to find problems. So if you are bored, you can also look at  all these in your own area. You can look for the errors in OpenStreet. There is a special  map where all the errors is pointed out. So you could just correct them if you are native  to that area. Okay. Okay. That would be the time to move to the next talk and let's  thank this speaker again. Thank you. Next talk will be in four minutes.
Anders_Lehmann_-_How_to_GIS_with_Python_0_2195.png Anders_Lehmann_-_How_to_GIS_with_Python_2196_2963.png Anders_Lehmann_-_How_to_GIS_with_Python_2964_3887.png Anders_Lehmann_-_How_to_GIS_with_Python_3888_4355.png Anders_Lehmann_-_How_to_GIS_with_Python_4356_4439.png Anders_Lehmann_-_How_to_GIS_with_Python_4440_4715.png Anders_Lehmann_-_How_to_GIS_with_Python_4716_4823.png Anders_Lehmann_-_How_to_GIS_with_Python_4824_8267.png Anders_Lehmann_-_How_to_GIS_with_Python_8268_8735.png Anders_Lehmann_-_How_to_GIS_with_Python_8736_14951.png Anders_Lehmann_-_How_to_GIS_with_Python_14952_16211.png Anders_Lehmann_-_How_to_GIS_with_Python_16212_16487.png Anders_Lehmann_-_How_to_GIS_with_Python_16488_16535.png Anders_Lehmann_-_How_to_GIS_with_Python_16536_16727.png Anders_Lehmann_-_How_to_GIS_with_Python_16728_16799.png Anders_Lehmann_-_How_to_GIS_with_Python_16800_16967.png Anders_Lehmann_-_How_to_GIS_with_Python_16968_17159.png Anders_Lehmann_-_How_to_GIS_with_Python_17160_17327.png Anders_Lehmann_-_How_to_GIS_with_Python_17328_17423.png Anders_Lehmann_-_How_to_GIS_with_Python_17424_17495.png Anders_Lehmann_-_How_to_GIS_with_Python_17496_18263.png Anders_Lehmann_-_How_to_GIS_with_Python_18264_25067.png Anders_Lehmann_-_How_to_GIS_with_Python_25068_25187.png Anders_Lehmann_-_How_to_GIS_with_Python_25188_25331.png Anders_Lehmann_-_How_to_GIS_with_Python_25332_25667.png Anders_Lehmann_-_How_to_GIS_with_Python_25668_26015.png Anders_Lehmann_-_How_to_GIS_with_Python_26016_28067.png Anders_Lehmann_-_How_to_GIS_with_Python_28068_31091.png Anders_Lehmann_-_How_to_GIS_with_Python_31092_31211.png Anders_Lehmann_-_How_to_GIS_with_Python_31212_31283.png Anders_Lehmann_-_How_to_GIS_with_Python_31284_49763.png Anders_Lehmann_-_How_to_GIS_with_Python_49764_51347.png Anders_Lehmann_-_How_to_GIS_with_Python_51348_52475.png Anders_Lehmann_-_How_to_GIS_with_Python_52476_52775.png Anders_Lehmann_-_How_to_GIS_with_Python_52776_54707.png Anders_Lehmann_-_How_to_GIS_with_Python_54708_57503.png Anders_Lehmann_-_How_to_GIS_with_Python_57504_59003.png Anders_Lehmann_-_How_to_GIS_with_Python_59004_59063.png Anders_Lehmann_-_How_to_GIS_with_Python_59064_59939.png Anders_Lehmann_-_How_to_GIS_with_Python_59940_59987.png Anders_Lehmann_-_How_to_GIS_with_Python_59988_60191.png Anders_Lehmann_-_How_to_GIS_with_Python_60192_60950.png