Pan-european seasonal cloudless mosaic based on Sentinel-2 imagery
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Križan, Josip
Josip Križan is the owner of MultiOne. In this talk, Josip introduced how the rapidly increasing amount of publically available remotely sensed data in recent decades has revolutionized large-scale research and context-informed decision making. However, these data are generally not freely available as homogenized products ready for analysis at continental (or larger) scales. This is widely observed with datasets generated by EO satellites, particularly those with optical sensors and those capable of high-resolution imaging, where the process of mosaicking imagery to produce a homogenous, cloudless dataset across a particular area of interest often grows increasingly cumbersome at larger scales.
 I'm Josep Križan. I come from Multivan Company. So we made this Pan-European seasonal cloudless mosaic with Sentinel-2 images. Can you see my screen? So there is a lot of publicly available remote-resense data, and it has been rapidly increasing in recent decades. But the data are not generally freely available as hominized and analysis-ready products. So the process of mosaicing these images to produce something that's homogeneous and cloudless can become a little bit cumbersome at larger scales. So this is the description of that as said. So the source of data were Sentinel-2 level 2A open data on AWS. We produced, we took the six bands from there, blue, green, red, nearly infrared sphere 1 sphere 2. We took, for every band, we took three statistics, 25, 50, 75 percentile and count of valid values, so per pixel. We produced four seasons per year in 2018 and 2019 and winter and spring for 2020. So these are dates when these dates are aligned with GLAAD IRD products. So that it was the idea that we made the fusion of this dataset with that Landsat product. And all data is a sample of 30 meters resolution in this projection in 2035. Okay. So data can be accessed in Zenodo platform. There is, you can download it from here. Or... Okay. Or there is also made the cloud optimized geotifs on Wasabi. So it can be accessed directly via web. It can be accessed directly via through Pantungis. This is the pattern of links. And so for band there is red, green, blue, near sphere 1, sphere 2. And for seasons, you have to replace this start date and end date in the links. If you want to access this with GLAAD tools, then you have to add prefix VCI curl. So this is example. How can you access this data? And yes, this data are, yes, and this data are scaled to bytes. Originally, data is in 16 bits. But just for a format of size. Size is very big. So you have to scale it to bytes. So this is how it looks. I can show you, I will show you in Pantungis. For example, this is summer 219. And this is autumn 219. It's not perfect, especially this autumn part. So how it's done? First, we calculate the mosaic of one season, one band in one centimeter tile and relative orbit. And then we just saw all the big values that has clouds or cloud shadows is masked out. And then it's calculated 25, 50, 75 percentile and count of all valid values for every pixel location. And this intermediate result is saved back to temporary S3 location. So this is done for all, for all, there was all 137 centi of two tiles. So this is, so this is a grid of centi of two tiles. So first it was every tile is calculated independently. And then, and then all tiles that are in the same season band and orbit is stitched together. It's just taking mean value of overlapping, not tiles, overlapping pixels. So this looks, it looks like something like this. So we have these tiles, but we have these orbits. So everything that is in one orbit is consistent because satellite is, it's, all these images are taken one after another. So there is no visible difference between them. But when we put neighboring orbit one by another, then we have see this that we have, they're not aligned perfectly. So, so this was these two orbits are then stitched together by using inverse distance weight mean with respect to the distances, distances from all from every pixel to to this. So, that way, when we put this all orbits together, we can get we have this final product where there is no stripes, there is no visible. Transitions from one orbit to another. Yes, this is. So, for example, winter 219 RGB, RGB composite looks something like this. There is, there is a lot of this white values, because there is just, it just snow, snow isn't, wasn't must out. And spring and summer. Maybe you could zoom into some area. Yeah. Yes, I can zoom it. So this is directly from wasabi. It's really fast. So this is. That's it. It's 30 30 meters resolution. And okay this is RGB composite. I made this near sphere composite to it looks interesting. This is. So this is from near sphere one and two bands. Okay. So, yes, that was our team that we used. And this was computed on multiple spot instances on the AWS. Every instance have 64 to send to the CPU send 500 to send 100 gigabytes of RAM. And so this one. So every season band style orbit mosaic is was one independent job. And it's, it could be calculated in parallel. So, there's a system is made of two components. And script that runs on this multiple instances, and it fetches job descriptions from on central server. And that server, there is a happy and behind this up is postgres database that keeps track of all jobs done all jobs that need to be done. And take care that there's no duplicate jobs and so on. And this approach is scalable to any number of instance with shoes with CPUs. So, of course, up to total number of jobs. It's not. Well, it's not really correct this scalable. If you have too many instances, then this central server becomes quite big. But okay, that that can be. So there is 137 s two tiles. And it was around 42 to 43,000 images sent in two images for a for each of the band for each season to make this, this all was like, it was really a lot of data. Well, that's, that's it. I was really quick. Okay, any questions. Okay, thank you. And so this data, can you go back to the nodo just show us one more time at the moment so it's available in the nodo and it will be put also in the open data viewer. And also at the moment is the value of choice. I don't have to download everything. But it will basically the data is ready. What about the gap feeling. Well, there's no, there's no, we didn't make any gap feeling. We have this account layers, where there is for every pixel there is a count of numbers so if it's lower number then this pixel is not really reliable. So it can be used like you get feeling but we didn't make any gap feeling at the moment for this. But it's a fantastic piece of work and we have both so Lancet and Sentinel. We have inside the project prepared analysis ready okay new video data that gaps with the Lancet we fill in all the gaps. What is your impression of the difference between Lancet and Sentinel, because there's a, you know, it's the same resolution is the kind of multi spectral same type of bands. So, impression so far about the differences, Lancet and Sentinel. Lancet has 16 days with revisiting time Sentinel has five days so it's almost three times more data on timeline. So that's the big deal. So there is less, less, less gaps for that. But then Lancet has this one's at seven months at five months at eight so that's you have, you have more than 20 years of Lancet, while Sentinel is from 216 17 so on.
 I'm Josep Križan.  I come from Multivan Company.  So we made this Pan-European seasonal cloudless mosaic with Sentinel-2 images.  Can you see my screen?  So there is a lot of publicly available remote-resense data, and it has been rapidly increasing in recent decades.  But the data are not generally freely available as hominized and analysis-ready products.  So the process of mosaicing these images to produce something that's homogeneous and cloudless can become a little bit cumbersome at larger scales.  So this is the description of that as said.  So the source of data were Sentinel-2 level 2A open data on AWS.  We produced, we took the six bands from there, blue, green, red, nearly infrared sphere 1 sphere 2.  We took, for every band, we took three statistics, 25, 50, 75 percentile and count of valid values, so per pixel.  We produced four seasons per year in 2018 and 2019 and winter and spring for 2020.  So these are dates when these dates are aligned with GLAAD IRD products.  So that it was the idea that we made the fusion of this dataset with that Landsat product.  And all data is a sample of 30 meters resolution in this projection in 2035.  Okay.  So data can be accessed in Zenodo platform.  There is, you can download it from here.  Or...  Okay.  Or there is also made the cloud optimized geotifs on Wasabi. So it can be accessed directly via web.  It can be accessed directly via through Pantungis.  This is the pattern of links.  And so for band there is red, green, blue, near sphere 1, sphere 2.  And for seasons, you have to replace this start date and end date in the links.  If you want to access this with GLAAD tools, then you have to add prefix VCI curl.  So this is example. How can you access this data?  And yes, this data are, yes, and this data are scaled to bytes.  Originally, data is in 16 bits.  But just for a format of size.  Size is very big. So you have to scale it to bytes.  So this is how it looks. I can show you, I will show you in Pantungis.  For example, this is summer 219. And this is autumn 219.  It's not perfect, especially this autumn part.  So how it's done?  First, we calculate the mosaic of one season, one band in one centimeter tile and relative orbit.  And then we just saw all the big values that has clouds or cloud shadows is masked out.  And then it's calculated 25, 50, 75 percentile and count of all valid values for every pixel location.  And this intermediate result is saved back to temporary S3 location.  So this is done for all, for all, there was all 137 centi of two tiles.  So this is, so this is a grid of centi of two tiles.  So first it was every tile is calculated independently.  And then, and then all tiles that are in the same season band and orbit is stitched together.  It's just taking mean value of overlapping, not tiles, overlapping pixels.  So this looks, it looks like something like this.  So we have these tiles, but we have these orbits.  So everything that is in one orbit is consistent because satellite is, it's, all these images are taken one after another.  So there is no visible difference between them. But when we put neighboring orbit one by another, then we have see this that we have,  they're not aligned perfectly.  So, so this was these two orbits are then stitched together by using inverse distance weight mean with respect to the distances, distances from all from every pixel to to this.  So, that way, when we put this all orbits together, we can get we have this final product where there is no stripes, there is no visible.  Transitions from one orbit to another.  Yes, this is.  So, for example, winter 219 RGB, RGB composite looks something like this.  There is, there is a lot of this white values, because there is just, it just snow, snow isn't, wasn't must out.  And spring and summer.  Maybe you could zoom into some area.  Yeah.  Yes, I can zoom it.  So this is directly from wasabi.  It's really fast.  So this is.  That's it. It's 30 30 meters resolution. And okay this is RGB composite. I made this near sphere composite to it looks interesting.  This is.  So this is from near sphere one and two bands.  Okay.  So, yes, that was our team that we used.  And this was computed on multiple spot instances on the AWS.  Every instance have 64 to send to the CPU send 500 to send 100 gigabytes of RAM.  And so this one.  So every season band style orbit mosaic is was one independent job. And it's, it could be calculated in parallel.  So, there's a system is made of two components.  And script that runs on this multiple instances, and it fetches job descriptions from on central server. And that server, there is a happy and behind this up is postgres database that keeps track of all jobs done all jobs that need to be done.  And take care that there's no duplicate jobs and so on.  And this approach is scalable to any number of instance with shoes with CPUs. So, of course, up to total number of jobs.  It's not. Well, it's not really correct this scalable. If you have too many instances, then this central server becomes quite big.  But okay, that that can be.  So there is 137 s two tiles.  And it was around 42 to 43,000 images sent in two images for a for each of the band for each season to make this, this all was like, it was really a lot of data.  Well, that's,  that's it.  I was really quick.  Okay, any questions.  Okay, thank you. And so this data, can you go back to the nodo just show us one more time at the moment so it's available in the nodo and it will be put also in the open data viewer.  And also at the moment is the value of choice. I don't have to download everything.  But it will basically the data is ready. What about the gap feeling.  Well, there's no, there's no, we didn't make any gap feeling.  We have this account layers, where there is for every pixel there is a count of numbers so if it's lower number then this pixel is not really reliable.  So it can be used like you get feeling but we didn't make any gap feeling at the moment for this.  But it's a fantastic piece of work and we have both so Lancet and Sentinel. We have inside the project prepared analysis ready okay new video data that gaps with the Lancet we fill in all the gaps.  What is your impression of the difference between Lancet and Sentinel, because there's a, you know, it's the same resolution is the kind of multi spectral same type of bands.  So, impression so far about the differences, Lancet and Sentinel.  Lancet has 16 days with revisiting time Sentinel has five days so it's almost three times more data on timeline. So that's the big deal. So there is less, less, less gaps for that.  But then Lancet has this one's at seven months at five months at eight so that's you have, you have more than 20 years of Lancet, while Sentinel is from 216 17 so on.
4_0_4523.png 4_4524_4667.png 4_4668_6563.png 4_6564_7415.png 4_7416_9815.png 4_9816_11363.png 4_11364_13139.png 4_13140_13439.png 4_13440_13487.png 4_13488_13631.png 4_13632_14627.png 4_14628_14831.png 4_14832_15359.png 4_15360_19055.png 4_19056_19211.png 4_19212_19307.png 4_19308_19739.png 4_19740_19907.png 4_19908_20027.png 4_20028_20807.png 4_20808_22894.png