PROBA-V mission exploitation platform
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Dries, Jeroen
In januari, the European Space Agency launched the first version of the PROBA-V mission exploitation platform. proba-v-mep.esa.int. This platform, which is fully operated by VITO Remote Sensing, has the goal to simplify the use of open remote sensing data which should eventually result in operational applications that benefit society. Exploitation platforms are the way of the future to handle the ever increasing volumes of remote sensing data, and at VITO we believe that the use of Open Source software is the only way to collaborate on this shared vision. In this talk, first I want to give a general overview on what users can do with the PROBA-V MEP. This involves using an Openstack VM loaded with FOSS software and direct access to the dataset, to access an Hadoop cluster where a user can distribute his processing using Spark. Secondly, I want to show how we are using Geotrellis to support interactive queries on the full timeseries of remote sensing data that is available in the platform. Also showing how this can be done from within an interactive Scala notebook in the browser.
 Okay, let's move to the last presentation. Thank you. Hi everybody. I'm Jeroen Driess from VITO presenting the mission exploitation platform for the ProbaV satellite mission which we created for ESA or which we are creating in fact. So I'll try to explain what it is a bit and give some demos of the application and try to explain as good as I can what the platform is. So we are VITO, basically the VITO remote sensing product unit and we cover the whole range of remote sensing applications. So we have our own platforms UAVs, Airborne, we are making sensors for those platforms and then we have a number of value added services that like creating products based on the data that we record. So our most important platform is in fact ProbaV. It's a satellite mission which captures daily a global image of 300 meter resolution and we are basically doing the user segment for that for ESA. So the data comes in each day. We do the processing, we do atmospheric correction and so on and we generate level three products that we distribute through our catalog service. And basically that is how a user segment has always been built up until now. So you just provide a catalog and a service where the user can download the data and he just downloads it from there and then he has to do whatever he wants, he can do whatever he wants with his own processing resources. Of course this means that there is also a long feedback loop between the users and the user segment. We also don't always know what they want or what they are doing with our data. So now also as those data sets become larger and larger, this downloading of data every time becomes a bit of an issue. So it's easier to bring the users to the data because we already have a fairly large data center where we host all of this. So if we simply add the processing resources then the user can just come to us and we provide a cloud platform where they can do their processing. Basically that's the vision of ESA for exploitation platforms. We're not the only one who's doing that. There will also be others. We are a mission exploitation platform because the mission is our central team but there is also thematic exploitation platforms like one for forestry or polar exploitation platform. Those also exist. So that's the ID. So what we have in there is of course this complete archive of the ProBavi satellite but also its predecessor SpotFigiti which provided one kilometer products. But there if you combine those two you'll get a time series which starts in 1998. So you get a pretty long time series for continuous monitoring of for instance vegetation. To that we add some scalable ICT resources using OpenStack so it's really a cloud environment that we have and we add some additional processing resources using the Hadoop distributed processing system which allows you to really do distributed processing, right, distributed algorithms on this data which is in this data center. So for instance a well-known tool is Spark. So you can write Spark jobs to all of the process each style of ProBavi. On top of that we also try to support you and try to provide documentation. So this exploitation platform is really a big thing which contains just a whole bunch of things. It's not just a software product or just a cloud platform. It's also the user support that comes with it of course. If you look at it from a technical side then it's basically a layered architecture where we have of course full archive of ProBavi but we can also provide access to other satellite missions because sometimes you want to combine data from various missions. So it makes sense to have a way to also query this data. So this ProBavi is of course online. It's very easy to reach, very fast, but we don't have the full Sentinel-2 dataset downloaded at our data center. Then on top of that we have those resources with OpenStack and Hadoop and then a whole bunch of OpenSars libraries. Almost everything we use is OpenSars so I won't be able to mention each project we use by name but it's the usual ones like Cheedal, also more cutting-edge stuff like Cheeltralis which can be used to do raster processing on top of Spark and Hadoop. I will show that later on. But also for instance you can very easily use Gras. And then using those libraries, of course you as a user will have direct access to, but we also built applications on top of that. And those applications is what I will try to demo right now. So the idea is actually we have some really end-user applications that are really accessible for a large group of users like if you're a journalist or if you don't know anything about remote sensing, about vegetation monitoring, you can still use them. But of course you have less flexibility. Then if you go for more flexibility we have iPython notebooks where a remote sensing expert who knows a bit of Python can use the algorithms that he knows, the Python modules that he knows to do processing, but also using Spark. So using these distributed resources to make things really go quite fast on this large dataset. And then if you go down all the way we also provide support for developers. So you will be able to plug in your own applications. Let's say if you have an algorithm and you want to provide it to the user but the algorithm requires a dataset, then you will be able to deploy your full service inside the platform. So you can really build your own applications. And we will we have some examples of that already. Well that we've built. So if we go to the website, this is it. It's probavmap.esa.int. It's a simple website. And then here we have the list of applications. So the first one, which is easiest to understand and which also shows the dataset is a simple viewer. So you can just view the global image, which we've recorded here since July, but it always contains the latest data and you can browse through this. So if I click it just it updates of course, you can zoom in. So of course, viewing is not something that a remote sensing expert does very often, but for a really more regular user, it's not an expert. This is already nice to have. We also have in here of course the S10 data, which is a 10 daily composite, which you can also view. So you can easily go through the dataset. Then of course, the next thing to do, or maybe I should say something else first, we have the probav data set, but we are also involved in the global land component of the Copernicus program. And for that, they also have some data products. For that we built similar services. So here if you go to the Copernicus global land service website, you can now access those viewing services. And for instance there, you can do the same thing, but for those products. And these are really products where you have sometimes a long time series of various vegetation in the chess that you can view on there. So that has also been released. It's basically the same application, but built for with the Copernicus products. And the next step is that we will just provide those layers also directly on the probav map website, which makes more sense. Then the next application, let me go back. So this was the viewer. Oh yeah, the design of the viewer is basically all open source. I won't go too deep into detail because there are much better GeoServer talks over here, but it is a cluster of GeoServers. And in front of it, it's a map cache server for the WMTS service for fast serving of web tiles. And then we have everything around it to do monitoring and so on. Of course, if you have any questions about that, I'll be here for all of the conference, so don't hesitate. Then the next demo is the time series viewer, which is more interesting because it provides access to pre-computed time series. As I said, for probav, the data starts in 2014, so it's a relatively short time series for Copernicus. It starts in 1998 sometimes. So really, if you want to do vegetation analysis, it's a nice tool to get a quick idea of differences between regions. Here it shows the time series at the country level on the map, but if you zoom in, which makes more sense to do, you can click on the regions and query the time series for a region and also for a specific land cover. That usually makes more sense to do because otherwise, if you aggregate over a whole bunch of entirely different land covers, you get just something which doesn't say as much. So this is, I mean, I was not a remote sensing expert myself when I started, but when you browse through these time series, you do see the differences between regions and you get to a good feel of what various parameters mean. Here we only have NDVI and you can combine it with rainfall, for instance, to get an idea of the correlation between NDVI and rainfall, of course. And if you go to the Copernicus version, you see that here the time series is much longer and also you have more parameters available. So if you combine the evapar with fcover, for instance, you can just get an idea of how they behave. We then the next step, because what people usually do in their research is that they have a very specific area of interest, that they are researching, they have certain parcels where they need to query the time series and that's what we will be doing next so that you can just draw your own polygon or maybe upload the shape file or click on a point and you get that time series on the fly. That's the next step. At this level it's still aggregated, which is not good enough for a lot of people, but then if you can really get it on the fly, you can really know for your own garden how it has been behaving over the last years. Only problem is the resolution is one kilometer, so you need a fairly big garden. That's the time series viewer application. So here, technology-wise, we build it using open layers, XGS, GeoX. There's a kind of OLAP cube behind it to serve the data, the time series data very fast and the time series themselves are extracted from the imagery using our Hadoop distributed processing environment. So it's basically all open source. This is the demo in case I don't have internet. Then the last application, which I will not really show is an example of how we want to provide algorithms to the community and an example of how you can integrate one algorithm. So a very easy example is on-demand compositing. We provide a 10-day composite with ProBV with a specific algorithm, but not everybody may be happy with that. Some users want to use this or that algorithm or want to use five days instead of 10 days, whatever. Using this tool, you can provide all of those parameters, select your area of interest and generate a composite. And the next step, which you can also do is create a subscription. So you say, okay, whenever new data becomes available, I want to have this new composite. And that's an example of an application that you could also write yourself as a developer. If you have your algorithm that you want to integrate, you can just deploy it on the cloud platform, potentially even write a UI for it like this, and then other people have access to your new invention. This is if you want to use this one, you need to log in on the portal. It's free to use. But if you create a login and you register, then you can actually do this. But basically, it's a matter of filling in the parameters. And a bit later, you will get your product. And the next step, if we go to remote sensing experts who really need flexibility, is what we provide is a toolbox with a bunch of open source tools available. So it's a regular VM on the cloud, like you can get on Amazon, but with a number of tools pre-installed, QJS, Cross, everything, and with direct access to the data archive on disk, so plain access. This will become normally available in Altem. We will try to somewhere end of September, do a release for this. It's not yet available. What is available right now is a toolbox on another cloud with a sample data set. But it's similar. So here you see how we just pre-installed the tool so you don't need to worry about that anymore. Then finally, the one application I also really like are the notebooks. They are very popular nowadays as a way to do reproducible research. So instead of writing a paper or complementary to that, people can write notebooks to publish their results, which are basically just bits of code and text and the results all in one thing. When it's interactive, let me just switch to the notebooks or wait. So I have one example where we analyzed the time series, which we have for the time series viewer. So the pre-computed time series for all the world. They're an interesting data set to work with if you want to do trend analysis. And then what you get out of it in the end is that you find regions in the world where there is a clear upward or downward trend. Now I have to add a disclaimer that this was not really done as a published result or a peer reviewed or whatever. So it's just an example of what can be done. So like for instance here at the bottom, you just see that for the downward trend, we get just some noisy data sets, which probably don't mean a lot. But if you guys probably also want to see the code, let me see where I can show that. Because here for instance, we use Spark. Here it is. So here is how you've loaded your data in a distributed data set and you can filter on it because the notebook is connected to the Hadoop cluster. So you can really do a powerful processing on that. Also machine learning for instance. And then the final demo I want to show is with GeoTrailers. This is a Scala notebook because GeoTrailers is written in Scala and the Jupyter Notebooks server also supports Scala kernels. The only thing we did not get to work here was the connection with the Hadoop cluster. So when we process here, it's on the Jupyter Hub server itself. And this because the Notebooks application will only be released somewhere around January next week. So this is really, really a prototype where some things just need to be worked out still. So here I really show how you can really just load a bunch of open source libraries in your notebook using Maven identifiers. You add them to your class path. Then you do some Spark initialization. This can all be copy pasted so you don't really need to be a real expert in that. And then you can start using this GeoTrailers library. Loading the data is very simple because we preloaded it into an accumulo database, which is a key value store on top of Hadoop. So we benefit from data locality. So you don't need to worry about how do I open my files, etc. It's just already in there. So you write this query where you specify your time of interest and your extent. And then you get this data set containing all of your tiles. Then if we go a bit further, I will post this on the website so that you can have a look at a later time. So here you see how you have these RDDs, which is a Spark concept, which is basically a distributed collection of something. You have those. And then finally what GeoTrails provides on that is a method to compute statistics. So here I define my function to compute mean values. And here it's just one method call basically. Well, once you have, you need to provide a mask and put it in the right format. So there are some code in between, of course. But once you have that, it's just one method call to compute stats for specific zones provided in a mask over your time series of data. And then because it wouldn't be a notebook if we could not plot it. So we also always need to have some nice plots. So eventually I could run it on three different data sets. So an NDEVI or two different data sets, NDEVI at 300 meter and the evapar from Copernicus at one kilometer. Yeah, here you kind of see the results, but it is not really about the results here. It's more about trying to show the technology and what a real expert could do with that. Good. I think I need to conclude. Yeah, there's a development that's still ongoing. That's the most important message here. And actually currently it's still a pre-release. So in autumn it will be the real first release, let's say. We're always looking for further collaborations because the more people we can get on the platform, the more use cases we can get to make it really work. This is the website. Yeah, no, this is the same thing. Okay, thank you. Any questions? Yeah, with the Spark, the book that you had, how much data was behind it? The last one was only a bit of data. Yeah, I could have shown you how I actually could run a cell. It works. And it doesn't take so long because I use a very small mask. Right, I was curious, have you tried scaling it? Yeah, that's the reason why I want to have the connection with Hadoop working. Right, so that's just the shortfall in the geotrends? The problem was in the combination, getting the Scala kernel of Jupiter Hub to work with geotrall and with Spark, but there are also other notebooks, alternatives available like Apache Zeppelin, for instance, that already integrates Spark and Hadoop directly. So there it works for sure. So we are still looking at what direction to go. Will we get this to work eventually? Or do we just have to go for another thing? I think the problem was a silly dependency issue because if you run code locally, it's simple. If you distribute code, then you get conflicts. If you already have certain classes on the cluster with a different version and so on. So that's the kind of thing you have to fix. The very quick issue is that you also can visualize dependency and label data. Is it also possible in your development environment to do this analysis between the dependencies that you're doing for and the library chunk? Yeah, well, you could do two things of cards. You have the time series data, the extracted time series are available on the platform. So then you're working on the aggregated data, but there you can do statistics between the rainfall and the NVI, for instance. And the other thing is, of course, that the rainfall imagery is also available on the platform. So, yeah, and from there you can do whatever you want, of course. Yeah. So we're working on an application where we're combining the probe of V and NVI values with that from MODIS combined. There's similar resolution. I'm wondering if any work has been done on validation or comparison between the NVI generator, P and other stuff. Not sure if I can answer that, I'm not the expert. I wouldn't say no, because there has been some validation between sensors, I think, but that's all I can say. Yeah. And the notebook is a lot scala. We also have Python notebooks, because we really try to, we know that a lot of people in the remote sensing community use Python, so you have to support that. But for instance, GeoTrails is written in scala, it doesn't have Python bindings, but it's also a very advanced interface. So yeah, we're looking, ideally we could provide Python bindings for that, but that's really something to work out over the coming years. Really from this, you could go to some kind of Google Earth Engine-like interface, maybe, if we work a lot, with less resources, of course, and then provide something in a number of languages. So you were saying you need to, you know, put all of your data in a similar way before processing something yourself? It's for the notebooks. So we, not all algorithms, use the Acumulo database, because also not all data is in there. All of the data is on disk in the regular GeoTIF-AGL5 formats. And then for specific data sets, we load them into Acumulo, because it is a duplication of data, so you have to, yeah, you need the resources for that, and you cannot just put any data set in there. It comes at a cost. But once it is in there, because right now we're thinking about putting most of the S10 composites in there, because it's very interesting to do interactive time series analysis in Acumulo. So and the S10s are smaller than daily products, because it's only every 10 days. So yeah. Did you find it was fairly reliable, and did you look at it through an ASAP? Yes, I know GeoMesa, but I haven't really dug into it. It's kind of similar. It would be interesting to also try that. And for now, the reliability of Acumulo, we have had some issues with versions, but we think it will work out, actually. It should be, I mean, it's quite stable. But we have really been working for the past months, let's say, on getting the Hadoop cluster running, on getting it secured, everything. So yeah, we do face a lot of issues everywhere, basically. Is there access to the data for free, and do you charge for processing? How we will charge? Right now it's still for free. Because it's an ESA project, we developed it for ESA, and we have resources to support a number of users, but it's not Google Earth Engine. We don't have unlimited resources. And at some point, also, when you start to do a lot of processing, then we may have to think about how to set up something where you pay per user or something, a model line. This still needs to be figured out what will be offered for free and what has to be paid for it. I would say don't hesitate to use it, because when it's not used at all, because you think that you might have to pay at some point and it won't get anywhere. So its first step is to get users on it. We have to think about a sustainable thing to keep it running, of course. Somebody has to come up with a bit of money. How to become a user? You can just register on the website. Right now, what you get when registering is mostly the end-dayly compositor application and this toolbox with sample data. So not that much. In autumn, we will do our first release. It will be on the website, but if you are a user, you might also get an email. And then you will get more available, like the full toolbox VM with access to the Hadoop cluster normally. And then notebooks will be at a later point. It's really step by step. You mentioned an old queue for a fancy. It's a combination of Cycu to translate MDX queries into SQL queries and Monete B for the SQL database. It's a columnar database, which is quite fast. Cycu is a business intelligence tool, basically. It's from that space. Yeah, your client's here in school. Do you have that access to that field? It's the web app. It has this Cycu API, in fact. So it could be offered more flexibly, although we didn't really think of doing that just yet. It could be done. And also what we do is put the data on the Hadoop platform. It's in a target file format, so an efficient file format, which you can also process easily using Spark. So then you can just work on there with very fast access to the full, all of the time series into whatever you want. Yep. Do you use a WPS or wanting access to a code? I think the compositor application is a WPS application. It's also the idea to use it more in the future for other interfaces. Or also when a developer implements his own algorithm and wants to expose it, we would like to expose it through the WPS standard. Maybe build an aggregated WPS, which combines all of the algorithms that have been developed and make it very easy also for developers to register their new thing, their new algorithm, and that they don't have to worry themselves too much about how to expose it as a WPS. Okay. Thank you. Yeah, thank you. Okay. Thank you. Thank you. Okay. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you for joining us today. Hi. Hi. It's самое. Great.
 Okay, let's move to the last presentation.  Thank you.  Hi everybody.  I'm Jeroen Driess from VITO presenting the mission exploitation platform for the ProbaV  satellite mission which we created for ESA or which we are creating in fact.  So I'll try to explain what it is a bit and give some demos of the application and try  to explain as good as I can what the platform is.  So we are VITO, basically the VITO remote sensing product unit and we cover the whole  range of remote sensing applications.  So we have our own platforms UAVs, Airborne, we are making sensors for those platforms  and then we have a number of value added services that like creating products based on the data  that we record.  So our most important platform is in fact ProbaV.  It's a satellite mission which captures daily a global image of 300 meter resolution and  we are basically doing the user segment for that for ESA.  So the data comes in each day.  We do the processing, we do atmospheric correction and so on and we generate level three products  that we distribute through our catalog service.  And basically that is how a user segment has always been built up until now.  So you just provide a catalog and a service where the user can download the data and he  just downloads it from there and then he has to do whatever he wants, he can do whatever  he wants with his own processing resources.  Of course this means that there is also a long feedback loop between the users and the  user segment.  We also don't always know what they want or what they are doing with our data.  So now also as those data sets become larger and larger, this downloading of data every  time becomes a bit of an issue.  So it's easier to bring the users to the data because we already have a fairly large data  center where we host all of this.  So if we simply add the processing resources then the user can just come to us and we provide  a cloud platform where they can do their processing.  Basically that's the vision of ESA for exploitation platforms.  We're not the only one who's doing that.  There will also be others.  We are a mission exploitation platform because the mission is our central team but there  is also thematic exploitation platforms like one for forestry or polar exploitation platform.  Those also exist.  So that's the ID.  So what we have in there is of course this complete archive of the ProBavi satellite but  also its predecessor SpotFigiti which provided one kilometer products.  But there if you combine those two you'll get a time series which starts in 1998.  So you get a pretty long time series for continuous monitoring of for instance vegetation.  To that we add some scalable ICT resources using OpenStack so it's really a cloud environment  that we have and we add some additional processing resources using the Hadoop distributed processing  system which allows you to really do distributed processing, right, distributed algorithms on  this data which is in this data center.  So for instance a well-known tool is Spark.  So you can write Spark jobs to all of the process each style of ProBavi.  On top of that we also try to support you and try to provide documentation.  So this exploitation platform is really a big thing which contains just a whole bunch  of things.  It's not just a software product or just a cloud platform.  It's also the user support that comes with it of course.  If you look at it from a technical side then it's basically a layered architecture where  we have of course full archive of ProBavi but we can also provide access to other satellite  missions because sometimes you want to combine data from various missions.  So it makes sense to have a way to also query this data.  So this ProBavi is of course online.  It's very easy to reach, very fast, but we don't have the full Sentinel-2 dataset downloaded  at our data center.  Then on top of that we have those resources with OpenStack and Hadoop and then a whole  bunch of OpenSars libraries.  Almost everything we use is OpenSars so I won't be able to mention each project we use  by name but it's the usual ones like Cheedal, also more cutting-edge stuff like Cheeltralis  which can be used to do raster processing on top of Spark and Hadoop.  I will show that later on.  But also for instance you can very easily use Gras.  And then using those libraries, of course you as a user will have direct access to, but  we also built applications on top of that.  And those applications is what I will try to demo right now.  So the idea is actually we have some really end-user applications that are really accessible  for a large group of users like if you're a journalist or if you don't know anything  about remote sensing, about vegetation monitoring, you can still use them.  But of course you have less flexibility.  Then if you go for more flexibility we have iPython notebooks where a remote sensing expert  who knows a bit of Python can use the algorithms that he knows, the Python modules that he  knows to do processing, but also using Spark.  So using these distributed resources to make things really go quite fast on this large  dataset.  And then if you go down all the way we also provide support for developers.  So you will be able to plug in your own applications.  Let's say if you have an algorithm and you want to provide it to the user but the algorithm  requires a dataset, then you will be able to deploy your full service inside the platform.  So you can really build your own applications.  And we will we have some examples of that already.  Well that we've built.  So if we go to the website, this is it.  It's probavmap.esa.int.  It's a simple website.  And then here we have the list of applications.  So the first one, which is easiest to understand and which also shows the dataset is a simple  viewer.  So you can just view the global image, which we've recorded here since July, but it always  contains the latest data and you can browse through this.  So if I click it just it updates of course, you can zoom in.  So of course, viewing is not something that a remote sensing expert does very often, but  for a really more regular user, it's not an expert.  This is already nice to have.  We also have in here of course the S10 data, which is a 10 daily composite, which you can  also view.  So you can easily go through the dataset.  Then of course, the next thing to do, or maybe I should say something else first, we have  the probav data set, but we are also involved in the global land component of the Copernicus  program.  And for that, they also have some data products.  For that we built similar services.  So here if you go to the Copernicus global land service website, you can now access  those viewing services.  And for instance there, you can do the same thing, but for those products.  And these are really products where you have sometimes a long time series of various vegetation  in the chess that you can view on there.  So that has also been released.  It's basically the same application, but built for with the Copernicus products.  And the next step is that we will just provide those layers also directly on the probav map  website, which makes more sense.  Then the next application, let me go back.  So this was the viewer.  Oh yeah, the design of the viewer is basically all open source.  I won't go too deep into detail because there are much better GeoServer talks over here,  but it is a cluster of GeoServers.  And in front of it, it's a map cache server for the WMTS service for fast serving of web  tiles.  And then we have everything around it to do monitoring and so on.  Of course, if you have any questions about that, I'll be here for all of the conference,  so don't hesitate.  Then the next demo is the time series viewer, which is more interesting because it provides  access to pre-computed time series.  As I said, for probav, the data starts in 2014, so it's a relatively short time series  for Copernicus.  It starts in 1998 sometimes.  So really, if you want to do vegetation analysis, it's a nice tool to get a quick idea of differences  between regions.  Here it shows the time series at the country level on the map, but if you zoom in, which  makes more sense to do, you can click on the regions and query the time series for a region  and also for a specific land cover.  That usually makes more sense to do because otherwise, if you aggregate over a whole bunch  of entirely different land covers, you get just something which doesn't say as much.  So this is, I mean, I was not a remote sensing expert myself when I started, but when you  browse through these time series, you do see the differences between regions and you get  to a good feel of what various parameters mean.  Here we only have NDVI and you can combine it with rainfall, for instance, to get an  idea of the correlation between NDVI and rainfall, of course.  And if you go to the Copernicus version, you see that here the time series is much longer  and also you have more parameters available.  So if you combine the evapar with fcover, for instance, you can just get an idea of  how they behave.  We then the next step, because what people usually do in their research is that they  have a very specific area of interest, that they are researching, they have certain parcels  where they need to query the time series and that's what we will be doing next so that  you can just draw your own polygon or maybe upload the shape file or click on a point and  you get that time series on the fly.  That's the next step.  At this level it's still aggregated, which is not good enough for a lot of people, but  then if you can really get it on the fly, you can really know for your own garden how  it has been behaving over the last years.  Only problem is the resolution is one kilometer, so you need a fairly big garden.  That's the time series viewer application.  So here, technology-wise, we build it using open layers, XGS, GeoX.  There's a kind of OLAP cube behind it to serve the data, the time series data very fast and  the time series themselves are extracted from the imagery using our Hadoop distributed  processing environment.  So it's basically all open source.  This is the demo in case I don't have internet.  Then the last application, which I will not really show is an example of how we want to  provide algorithms to the community and an example of how you can integrate one algorithm.  So a very easy example is on-demand compositing.  We provide a 10-day composite with ProBV with a specific algorithm, but not everybody may  be happy with that.  Some users want to use this or that algorithm or want to use five days instead of 10 days,  whatever.  Using this tool, you can provide all of those parameters, select your area of interest and  generate a composite.  And the next step, which you can also do is create a subscription.  So you say, okay, whenever new data becomes available, I want to have this new composite.  And that's an example of an application that you could also write yourself as a developer.  If you have your algorithm that you want to integrate, you can just deploy it on the  cloud platform, potentially even write a UI for it like this, and then other people have  access to your new invention.  This is if you want to use this one, you need to log in on the portal.  It's free to use.  But if you create a login and you register, then you can actually do this.  But basically, it's a matter of filling in the parameters.  And a bit later, you will get your product.  And the next step, if we go to remote sensing experts who really need flexibility, is what  we provide is a toolbox with a bunch of open source tools available.  So it's a regular VM on the cloud, like you can get on Amazon, but with a number of tools  pre-installed, QJS, Cross, everything, and with direct access to the data archive on  disk, so plain access.  This will become normally available in Altem.  We will try to somewhere end of September, do a release for this.  It's not yet available.  What is available right now is a toolbox on another cloud with a sample data set.  But it's similar.  So here you see how we just pre-installed the tool so you don't need to worry about  that anymore.  Then finally, the one application I also really like are the notebooks.  They are very popular nowadays as a way to do reproducible research.  So instead of writing a paper or complementary to that, people can write notebooks to publish  their results, which are basically just bits of code and text and the results all in one  thing.  When it's interactive, let me just switch to the notebooks or wait.  So I have one example where we analyzed the time series, which we have for the time series  viewer.  So the pre-computed time series for all the world.  They're an interesting data set to work with if you want to do trend analysis.  And then what you get out of it in the end is that you find regions in the world where  there is a clear upward or downward trend.  Now I have to add a disclaimer that this was not really done as a published result or a  peer reviewed or whatever.  So it's just an example of what can be done.  So like for instance here at the bottom, you just see that for the downward trend, we get  just some noisy data sets, which probably don't mean a lot.  But if you guys probably also want to see the code, let me see where I can show that.  Because here for instance, we use Spark.  Here it is.  So here is how you've loaded your data in a distributed data set and you can filter  on it because the notebook is connected to the Hadoop cluster.  So you can really do a powerful processing on that.  Also machine learning for instance.  And then the final demo I want to show is with GeoTrailers.  This is a Scala notebook because GeoTrailers is written in Scala and the Jupyter Notebooks  server also supports Scala kernels.  The only thing we did not get to work here was the connection with the Hadoop cluster.  So when we process here, it's on the Jupyter Hub server itself.  And this because the Notebooks application will only be released somewhere around January  next week.  So this is really, really a prototype where some things just need to be worked out still.  So here I really show how you can really just load a bunch of open source libraries in your  notebook using Maven identifiers.  You add them to your class path.  Then you do some Spark initialization.  This can all be copy pasted so you don't really need to be a real expert in that.  And then you can start using this GeoTrailers library.  Loading the data is very simple because we preloaded it into an accumulo database, which  is a key value store on top of Hadoop.  So we benefit from data locality.  So you don't need to worry about how do I open my files, etc.  It's just already in there.  So you write this query where you specify your time of interest and your extent.  And then you get this data set containing all of your tiles.  Then if we go a bit further, I will post this on the website so that you can have a look  at a later time.  So here you see how you have these RDDs, which is a Spark concept, which is basically a distributed  collection of something.  You have those.  And then finally what GeoTrails provides on that is a method to compute statistics.  So here I define my function to compute mean values.  And here it's just one method call basically.  Well, once you have, you need to provide a mask and put it in the right format.  So there are some code in between, of course.  But once you have that, it's just one method call to compute stats for specific zones provided  in a mask over your time series of data.  And then because it wouldn't be a notebook if we could not plot it.  So we also always need to have some nice plots.  So eventually I could run it on three different data sets.  So an NDEVI or two different data sets, NDEVI at 300 meter and the evapar from Copernicus  at one kilometer.  Yeah, here you kind of see the results, but it is not really about the results here.  It's more about trying to show the technology and what a real expert could do with that.  Good.  I think I need to conclude.  Yeah, there's a development that's still ongoing.  That's the most important message here.  And actually currently it's still a pre-release.  So in autumn it will be the real first release, let's say.  We're always looking for further collaborations because the more people we can get on the  platform, the more use cases we can get to make it really work.  This is the website.  Yeah, no, this is the same thing.  Okay, thank you.  Any questions?  Yeah, with the Spark, the book that you had, how much data was behind it?  The last one was only a bit of data.  Yeah, I could have shown you how I actually could run a cell.  It works.  And it doesn't take so long because I use a very small mask.  Right, I was curious, have you tried scaling it?  Yeah, that's the reason why I want to have the connection with Hadoop working.  Right, so that's just the shortfall in the geotrends?  The problem was in the combination, getting the Scala kernel of Jupiter Hub to work with  geotrall and with Spark, but there are also other notebooks, alternatives available like  Apache Zeppelin, for instance, that already integrates Spark and Hadoop directly.  So there it works for sure.  So we are still looking at what direction to go.  Will we get this to work eventually?  Or do we just have to go for another thing?  I think the problem was a silly dependency issue because if you run code locally, it's  simple.  If you distribute code, then you get conflicts.  If you already have certain classes on the cluster with a different version and so on.  So that's the kind of thing you have to fix.  The very quick issue is that you also can visualize dependency and label data.  Is it also possible in your development environment to do this analysis between the dependencies  that you're doing for and the library chunk?  Yeah, well, you could do two things of cards.  You have the time series data, the extracted time series are available on the platform.  So then you're working on the aggregated data, but there you can do statistics between the  rainfall and the NVI, for instance.  And the other thing is, of course, that the rainfall imagery is also available on the  platform.  So, yeah, and from there you can do whatever you want, of course.  Yeah.  So we're working on an application where we're combining the probe of V and NVI values  with that from MODIS combined.  There's similar resolution.  I'm wondering if any work has been done on validation or comparison between the NVI  generator, P and other stuff.  Not sure if I can answer that, I'm not the expert.  I wouldn't say no, because there has been some validation between sensors, I think,  but that's all I can say.  Yeah.  And the notebook is a lot scala.  We also have Python notebooks, because we really try to, we know that a lot of people  in the remote sensing community use Python, so you have to support that.  But for instance, GeoTrails is written in scala, it doesn't have Python bindings, but  it's also a very advanced interface.  So yeah, we're looking, ideally we could provide Python bindings for that, but that's  really something to work out over the coming years.  Really from this, you could go to some kind of Google Earth Engine-like interface, maybe,  if we work a lot, with less resources, of course, and then provide something in a number  of languages.  So you were saying you need to, you know, put all of your data in a similar way before  processing something yourself?  It's for the notebooks.  So we, not all algorithms, use the Acumulo database, because also not all data is in  there.  All of the data is on disk in the regular GeoTIF-AGL5 formats.  And then for specific data sets, we load them into Acumulo, because it is a duplication  of data, so you have to, yeah, you need the resources for that, and you cannot just put  any data set in there.  It comes at a cost.  But once it is in there, because right now we're thinking about putting most of the S10  composites in there, because it's very interesting to do interactive time series analysis in  Acumulo.  So and the S10s are smaller than daily products, because it's only every 10 days.  So yeah.  Did you find it was fairly reliable, and did you look at it through an ASAP?  Yes, I know GeoMesa, but I haven't really dug into it.  It's kind of similar.  It would be interesting to also try that.  And for now, the reliability of Acumulo, we have had some issues with versions, but we  think it will work out, actually.  It should be, I mean, it's quite stable.  But we have really been working for the past months, let's say, on getting the Hadoop cluster  running, on getting it secured, everything.  So yeah, we do face a lot of issues everywhere, basically.  Is there access to the data for free, and do you charge for processing?  How we will charge?  Right now it's still for free.  Because it's an ESA project, we developed it for ESA, and we have resources to support  a number of users, but it's not Google Earth Engine.  We don't have unlimited resources.  And at some point, also, when you start to do a lot of processing, then we may have to  think about how to set up something where you pay per user or something, a model line.  This still needs to be figured out what will be offered for free and what has to be paid  for it.  I would say don't hesitate to use it, because when it's not used at all, because you think  that you might have to pay at some point and it won't get anywhere.  So its first step is to get users on it.  We have to think about a sustainable thing to keep it running, of course.  Somebody has to come up with a bit of money.  How to become a user?  You can just register on the website.  Right now, what you get when registering is mostly the end-dayly compositor application  and this toolbox with sample data.  So not that much.  In autumn, we will do our first release.  It will be on the website, but if you are a user, you might also get an email.  And then you will get more available, like the full toolbox VM with access to the Hadoop  cluster normally.  And then notebooks will be at a later point.  It's really step by step.  You mentioned an old queue for a fancy.  It's a combination of Cycu to translate MDX queries into SQL queries and Monete B for  the SQL database.  It's a columnar database, which is quite fast.  Cycu is a business intelligence tool, basically.  It's from that space.  Yeah, your client's here in school.  Do you have that access to that field?  It's the web app.  It has this Cycu API, in fact.  So it could be offered more flexibly, although we didn't really think of doing that just  yet.  It could be done.  And also what we do is put the data on the Hadoop platform.  It's in a target file format, so an efficient file format, which you can also process easily  using Spark.  So then you can just work on there with very fast access to the full, all of the time series  into whatever you want.  Yep.  Do you use a WPS or wanting access to a code?  I think the compositor application is a WPS application.  It's also the idea to use it more in the future for other interfaces.  Or also when a developer implements his own algorithm and wants to expose it, we would  like to expose it through the WPS standard.  Maybe build an aggregated WPS, which combines all of the algorithms that have been developed  and make it very easy also for developers to register their new thing, their new algorithm,  and that they don't have to worry themselves too much about how to expose it as a WPS.  Okay.  Thank you.  Yeah, thank you.  Okay.  Thank you.  Thank you.  Okay.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you for joining us today.  Hi.  Hi.  It's самое.  Great.
foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_0_971.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_972_2471.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_2472_9671.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_9672_11867.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_11868_13391.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_13392_14483.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_14484_15179.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_15180_15599.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_15600_15743.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_15744_17231.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_17232_17891.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_17892_18827.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_18828_19571.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_19572_19919.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_19920_20723.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_20724_21059.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_21060_27023.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_27024_27719.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_27720_28187.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_28188_28271.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_28272_28991.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_28992_29423.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_29424_30467.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_30468_34895.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_34896_35051.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_35052_43763.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_43764_48251.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_48252_48299.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_48300_49127.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_49128_49211.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_49212_49271.png foss4g-2016-1362-proba-v_mission_exploitation_platform-hd_49272_49447.png