Development of a new framework for Distributed Processing of Big Geospatial Data
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Olasz, Angéla (Department of Geoinformation, Institute of Geodesy, Cartography and Remote Sensing (FÖMI),)
Kristof, Daniel (FOMI - Institute of Geodesy, Cartography and Remote Sensing)
The Geospatial world is still facing the lack of well-established distributed processing solutions tailored to the amount and heterogeneity of geodata, especially when fast data processing is a must. However, most current distributed computing frameworks have important limitations regarding both data distribution and data partitioning methods. Hence, this paper presents a prototype for tiling, stitching and processing of big geospatial data. The system is based on the IQLib concept developed in the frame of the IQmulus EU FP7 research and development project. The data distribution framework has no limitations on programming language environment and can execute scripts (and workflows) written in different development frameworks (e.g. Python, R or C#). It is capable of processing raster, vector and point cloud data. Our intention is to provide a solution to perform a wide range of geospatial processing capabilities in a distributed environment with no restrictions on data storage concepts. Our research covers methods controlling data partitioning, distributed processing and data assimilation as well. Partitioning (also referred to as “Tiling”) is a very delicate yet crucial step having impact on the whole processing. After algorithms have processed these “chunks” or “tiles” of data, partial results are collected to carry out data assimilation or “Stitching”. The paper presents the above-mentioned prototype through a case study dealing with country-wide processing of raster imagery. Assessment is carried out by comparing the results (computing time, accuracy, etc.) to concurrent solutions. Further investigations on algorithmic and implementation details are in focus for the near future.
 Next session by Angela. Thank you. Okay, so my name is Angela Olaz. My co-author has been Nguyen Tai, and unfortunately he couldn't come to answer your questions, but later on you can contact us. So our research is about the development of a new framework for distributed processing of big geospatial data, and this is a joint research of the Institute of Geodesic Artography and Remote Sensing in short, name is FEMI, as you can see here. And the University, the Elta University, located in Budapest, Hungary. The content of this talk is presented in this slide, so I'm going to give you a short introduction about our research topic, and then I'm going to introduce a project called ICUMULUS, which is related to this work, but I'm not going to give you a detailed introduction on the project. And I'm going to continue trying to define what is geospatial big data, what are the differences to big data, which is not geospatial, and what are the differences to geospatial data, which is not big. So, and then I would like to show you some comparison of existing solutions that we have tried to compare some software or frameworks, how they are doing now the distributed processing, and I'm going to present also what kind of user requirements that we have selected to compare those solutions. And then I'm going to present IQlib, which is the main development of our research, and its modular structure. It has a modular structure, so I'm going to present the modules and the actual development status of those modules. And I'm going to conclude in the last slides and some thoughts about the future work. So, introduction, our goal is to find a solution for processing big geospatial data in the distributed ecosystem without any limitations on programming language, as well as data partitioning and data distribution among the nodes, and in order to run existing GIS processing scripts. As a first step, we focus on raster data representation, for example, the composing those data sets and then distributed processing. Before building this prototype system, we have analyzed the data decomposition patterns, how can the raster data set can be decomposed and then processed on the different nodes, and then defined the common GIS user requirements on the processing environments for big geospatial data. So, we have some user requirements that they would think that is important for framework or toolkit that is supporting distributed processing and also identifying the geospatial big data. Some thoughts about the ICUMULUS project is this research is related to the ICUMULUS, which is about how volume fusion and analysis platform for geospatial point clouds, coverages and volumetric data sets. So, this is the main goal of the project, so it's a platform. This project is going to be finished in this November. As a result, we are going to have this analysis platform, so as we have tried to define everything on this platform, it can be available. So, we are in the consortium, which is formed by 11 European partner institutions, and we are from Hungary. And if you want to have more information on ICUMULUS project, please visit the website, ICUMULUS.eu. So, defining geospatial big data is not an easy task. It is the well-known definition where you can start to exceed the capability or the capacity of the current computing background of your available system or your available technology. And there are others in the literature that you can find that there is quite a big number of variables because why it's not so easy to define what is the margin between geospatial big data and geospatial data. So, some of them are also admitting that is also a use case specific. So, it is also hard to define what is big data for one user and what is for another. So, we have tried to compare big data, which is not geospatial, and geospatial big data and geospatial data as a short review from the three kind of data representation, data format for vector raster and 3D representation, and then also compared those to non-geospatial or text-based data format. And in this paper, where it maybe is going to be published until now, I don't know, we have collected those aspects for these three main theory or main definition in the representation and also in the storage and processing background of the requirements as you wish. And then this table is continuous where we included the existing solutions on those formats for each of these three definitions. And some requirements that would be useful to have from an existing solution. So, we have collected the most popular framework supporting distributed computing on GIS data. And for example, we have selected the following aspects, which is also then listed and included in a table and made a comparison between them. So, we would admit that input and output data types are of course important, what kind of data they are supporting. If already existing GIS processing or executables are supported to run or not, this is the main point, one of the main point, and what kind of data management they are supporting. Supervision of the data distribution and especially for the raster data type. So, we would like to have full control of what data chunks are going to which node and then getting back the process data and other aspects like scalability potential and supported platform and so on. So, we have collected all of those informations and tried to compare the existing solutions. And this table is already been published in a paper and then in the later slides I'm going to present where you can find. So, after all, and also from an experience from the ICUM-Linus project, we thought and we admitted that most of the cases that is the full control over our data partitioning and data distribution mechanism is not supported. So, and also it's not really possible to run already existing executables or scripts in a platform or in an ecosystem. So, we decided to develop our own distributed processing framework. And this has been initialized by three project partners. One is Aegean France, the second is CNR, Imaati, Genova and Fermi from Hungary. The name is IQlib and this IQlib is going to be a framework that is supporting the data decomposition as core functionality. This is called tiling and then data distribution and distributed data processing. The second has a second main functionality and then IQlib is also providing a functionality to stitch those results. So, this can cause that we can overcome the scalability limitations of the processing algorithms. So, the high-level concept is formed by this way. This is already a bit, could be updated because there is a new module that I'm going to introduce in the next slide. So, the main thing is there is the tiling and stitching part, the data distribution part that we would like to apply and then the already existing processing remote sensing or GIS scripts can be run on those datasets. As a researcher from the National Mapping Agency, this would be very useful because we have already developed our operational GIS processing in a different system or in a different language. So, this would be very useful if we can have a framework that this, that this can be somehow used for the processing in a distributed way. So, those are the modules of the IQlib. There is a data catalog module which is responsible for storing the metadata and also not only data about the data but also data about the processing. So, we would like to have also information on the data chunks and then also on the results and those are going to be stored in the data catalog module. So, there is a tiling and stitching module which is responsible to, to tile and to stitch. So, the metadata of the tile and also the stitch data is going to be also in the data catalog as I mentioned before. So, there is this new module that is responsible for the data distribution and there is also the processing module that is responsible to run those scripts on the already distributed dataset. So, the status of those modules, the data catalog is almost ready, is only waiting for the final approval from all partners and then is going to be available in an open source way. And in the first figure you can see the architecture of the data catalog module and the used software and then the tiling and stitching module is already defined and also we have the second figure. So, here you can see the architecture, the high level concept of how it's going to work but this is still under planning phase and then the data distribution module and this is a new one and it is currently supporting SFTP protocol only. But the data partitioning and the data distribution algorithm could be extended by third party developers. So, if you would like to add some ideas, do not hesitate to do it. And then distributed processing module is also under development and you can see the figure on the figure that the architecture of this module looks like this. And all of this information can be found in GitHub. This is IQ Live specification but it is going to be presented as dedicated IQ Live GitHub as well. So, as soon as possible, as I know, but the specification and technical details are already there. Okay, so the related papers, we had some presentations already and papers in ISPRS about this topic but we need of course a lot of work to do and the future work. So, we would like to finish those implementations for all of these modules, testing IQ Live for the following aspects, running existing algorithms. And then experiment execution on big geospatial data and then benchmarking mainly on the processing time. So, thank you for your attention and I also would like to thank to IQumulus and to Daniel Christof for letting us to be here. Thank you. Thank you. Questions? Thank you very much for the presentation. That is very interesting. There's one thing that was not completely clear for me. So, the algorithm itself, it runs over a distributed dataset but is the algorithm distributed as well? For instance, if you have a sequential algorithm in R or something, the algorithm will still be non-distributed. If it's possible, as I mean, if it's possible by the algorithm, which is written as it is possible to be parallel, so it's possible of course. So, you support both? Yes, we would like to support those because without this, it's not possible to run. Thank you very much. Hi. When you distribute the computation, there is some form of load balancing that is involved inside this process or not? This I don't know. So, maybe we can contact my co-author, who is the developer, so he can ask for your question. One more question? Am I right that all data is tiled during loading to a system or not? If the user wants, yes. If it's not needed because we have enough power to make it not tiled away, it's not needed. And the vector data with tiled, can it be editable after this? I think it's possible, but I don't know because we are focusing more on raster datasets until now. So, the other project partners working more on the vectors. Thank you very much. What kind of algorithms do you think can be run in this framework? I think we can have, like I have shown something in this here, like R, Python, Java, and Matlab as I hope. But if you could add something, I would like to know. Well, I was thinking about the processing itself. I mean, if the input is tiled, then it has to be local somehow. It can be a global algorithm, for example. Yes, it depends on the algorithms that is something related to the first question. Possibly. Maybe I can answer this question because at Mapbox we have the same problem. We solved it similarly in having a MapReduce model. And one application that we use it for is finding missing streets in open street map, which is a local problem that is defined to one tile. And we use telemetry data to figure out, oh, a lot of people drive on the street, but it doesn't exist on the tile. More questions? Okay, thank you.
 Next session by Angela.  Thank you.  Okay, so my name is Angela Olaz.  My co-author has been Nguyen Tai,  and unfortunately he couldn't come to answer your questions,  but later on you can contact us.  So our research is about the development of a new framework  for distributed processing of big geospatial data,  and this is a joint research of the Institute of Geodesic Artography  and Remote Sensing in short, name is FEMI, as you can see here.  And the University, the Elta University, located in Budapest, Hungary.  The content of this talk is presented in this slide,  so I'm going to give you a short introduction about our research topic,  and then I'm going to introduce a project called ICUMULUS,  which is related to this work, but I'm not going to give you a detailed introduction on the project.  And I'm going to continue trying to define what is geospatial big data,  what are the differences to big data, which is not geospatial,  and what are the differences to geospatial data, which is not big.  So, and then I would like to show you some comparison of existing solutions  that we have tried to compare some software or frameworks,  how they are doing now the distributed processing,  and I'm going to present also what kind of user requirements that we have selected to compare those solutions.  And then I'm going to present IQlib, which is the main development of our research,  and its modular structure.  It has a modular structure, so I'm going to present the modules  and the actual development status of those modules.  And I'm going to conclude in the last slides and some thoughts about the future work.  So, introduction, our goal is to find a solution for processing big geospatial data in the distributed ecosystem  without any limitations on programming language, as well as data partitioning  and data distribution among the nodes, and in order to run existing GIS processing scripts.  As a first step, we focus on raster data representation,  for example, the composing those data sets and then distributed processing.  Before building this prototype system, we have analyzed the data decomposition patterns,  how can the raster data set can be decomposed and then processed on the different nodes,  and then defined the common GIS user requirements on the processing environments for big geospatial data.  So, we have some user requirements that they would think that is important for framework or toolkit  that is supporting distributed processing and also identifying the geospatial big data.  Some thoughts about the ICUMULUS project is this research is related to the ICUMULUS,  which is about how volume fusion and analysis platform for geospatial point clouds,  coverages and volumetric data sets.  So, this is the main goal of the project, so it's a platform.  This project is going to be finished in this November.  As a result, we are going to have this analysis platform,  so as we have tried to define everything on this platform, it can be available.  So, we are in the consortium, which is formed by 11 European partner institutions,  and we are from Hungary.  And if you want to have more information on ICUMULUS project, please visit the website,  ICUMULUS.eu.  So, defining geospatial big data is not an easy task.  It is the well-known definition where you can start to exceed the capability  or the capacity of the current computing background of your available system  or your available technology.  And there are others in the literature that you can find that there is quite a big number of variables  because why it's not so easy to define what is the margin between geospatial big data and geospatial data.  So, some of them are also admitting that is also a use case specific.  So, it is also hard to define what is big data for one user and what is for another.  So, we have tried to compare big data, which is not geospatial, and geospatial big data and geospatial data  as a short review from the three kind of data representation, data format for vector raster  and 3D representation, and then also compared those to non-geospatial or text-based data format.  And in this paper, where it maybe is going to be published until now, I don't know,  we have collected those aspects for these three main theory or main definition in the representation  and also in the storage and processing background of the requirements as you wish.  And then this table is continuous where we included the existing solutions on those formats  for each of these three definitions.  And some requirements that would be useful to have from an existing solution.  So, we have collected the most popular framework supporting distributed computing on GIS data.  And for example, we have selected the following aspects, which is also then listed and included in a table  and made a comparison between them.  So, we would admit that input and output data types are of course important,  what kind of data they are supporting.  If already existing GIS processing or executables are supported to run or not,  this is the main point, one of the main point, and what kind of data management they are supporting.  Supervision of the data distribution and especially for the raster data type.  So, we would like to have full control of what data chunks are going to which node  and then getting back the process data and other aspects like scalability potential  and supported platform and so on.  So, we have collected all of those informations and tried to compare the existing solutions.  And this table is already been published in a paper and then in the later slides I'm going to present where you can find.  So, after all, and also from an experience from the ICUM-Linus project, we thought and we admitted that most of the cases  that is the full control over our data partitioning and data distribution mechanism is not supported.  So, and also it's not really possible to run already existing executables or scripts in a platform or in an ecosystem.  So, we decided to develop our own distributed processing framework.  And this has been initialized by three project partners.  One is Aegean France, the second is CNR, Imaati, Genova and Fermi from Hungary.  The name is IQlib and this IQlib is going to be a framework that is supporting the data decomposition as core functionality.  This is called tiling and then data distribution and distributed data processing.  The second has a second main functionality and then IQlib is also providing a functionality to stitch those results.  So, this can cause that we can overcome the scalability limitations of the processing algorithms.  So, the high-level concept is formed by this way.  This is already a bit, could be updated because there is a new module that I'm going to introduce in the next slide.  So, the main thing is there is the tiling and stitching part, the data distribution part that we would like to apply and then the already existing processing remote sensing or GIS scripts can be run on those datasets.  As a researcher from the National Mapping Agency, this would be very useful because we have already developed our operational GIS processing in a different system or in a different language.  So, this would be very useful if we can have a framework that this, that this can be somehow used for the processing in a distributed way.  So, those are the modules of the IQlib.  There is a data catalog module which is responsible for storing the metadata and also not only data about the data but also data about the processing.  So, we would like to have also information on the data chunks and then also on the results and those are going to be stored in the data catalog module.  So, there is a tiling and stitching module which is responsible to, to tile and to stitch.  So, the metadata of the tile and also the stitch data is going to be also in the data catalog as I mentioned before.  So, there is this new module that is responsible for the data distribution and there is also the processing module that is responsible to run those scripts on the already distributed dataset.  So, the status of those modules, the data catalog is almost ready, is only waiting for the final approval from all partners and then is going to be available in an open source way.  And in the first figure you can see the architecture of the data catalog module and the used software and then the tiling and stitching module is already defined and also we have the second figure.  So, here you can see the architecture, the high level concept of how it's going to work but this is still under planning phase and then the data distribution module and this is a new one and it is currently supporting SFTP protocol only.  But the data partitioning and the data distribution algorithm could be extended by third party developers.  So, if you would like to add some ideas, do not hesitate to do it.  And then distributed processing module is also under development and you can see the figure on the figure that the architecture of this module looks like this.  And all of this information can be found in GitHub. This is IQ Live specification but it is going to be presented as dedicated IQ Live GitHub as well.  So, as soon as possible, as I know, but the specification and technical details are already there.  Okay, so the related papers, we had some presentations already and papers in ISPRS about this topic but we need of course a lot of work to do and the future work.  So, we would like to finish those implementations for all of these modules, testing IQ Live for the following aspects, running existing algorithms.  And then experiment execution on big geospatial data and then benchmarking mainly on the processing time.  So, thank you for your attention and I also would like to thank to IQumulus and to Daniel Christof for letting us to be here.  Thank you.  Thank you. Questions?  Thank you very much for the presentation. That is very interesting.  There's one thing that was not completely clear for me. So, the algorithm itself, it runs over a distributed dataset but is the algorithm distributed as well?  For instance, if you have a sequential algorithm in R or something, the algorithm will still be non-distributed.  If it's possible, as I mean, if it's possible by the algorithm, which is written as it is possible to be parallel, so it's possible of course.  So, you support both?  Yes, we would like to support those because without this, it's not possible to run.  Thank you very much.  Hi. When you distribute the computation, there is some form of load balancing that is involved inside this process or not?  This I don't know. So, maybe we can contact my co-author, who is the developer, so he can ask for your question.  One more question?  Am I right that all data is tiled during loading to a system or not?  If the user wants, yes.  If it's not needed because we have enough power to make it not tiled away, it's not needed.  And the vector data with tiled, can it be editable after this?  I think it's possible, but I don't know because we are focusing more on raster datasets until now.  So, the other project partners working more on the vectors.  Thank you very much.  What kind of algorithms do you think can be run in this framework?  I think we can have, like I have shown something in this here, like R, Python, Java, and Matlab as I hope.  But if you could add something, I would like to know.  Well, I was thinking about the processing itself. I mean, if the input is tiled, then it has to be local somehow.  It can be a global algorithm, for example.  Yes, it depends on the algorithms that is something related to the first question.  Possibly.  Maybe I can answer this question because at Mapbox we have the same problem.  We solved it similarly in having a MapReduce model.  And one application that we use it for is finding missing streets in open street map, which is a local problem that is defined to one tile.  And we use telemetry data to figure out, oh, a lot of people drive on the street, but it doesn't exist on the tile.  More questions?  Okay, thank you.
foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_0_575.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_576_635.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_636_1607.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_1608_4439.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_4440_10679.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_10680_13991.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_13992_14495.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_14496_17507.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_17508_20735.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_20736_21095.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_21096_23495.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_23496_25475.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_25476_25547.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_25548_25595.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_25596_25643.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_25644_25727.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_25728_26099.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_26100_26147.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_26148_26231.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_26232_26291.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_26292_26459.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_26460_26531.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_26532_26627.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_26628_28019.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_28020_29339.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_29340_31091.png foss4g-2016-1284-development_of_a_new_framework_for_distributed_processing_of_big_geospatial_data-hd_31092_31241.png