Multispectral WebGL
CC Attribution - NonCommercial - NoDerivatives 4.0 International:
You are free to use, copy, distribute and transmit the work or content in unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Ortega Sánchez, Iván
How graphics tech (WebGL, texture formats and shaders) and multispectral imagery (GeoTIFFs with multiple bands or samples) are a perfect match. An overview on the work done in Leaflet.TileLayer.GL (and its OpenLayers port) to let web applications consume raw GeoTIFF data (including multiple bands and pixel formats), in order to perform raster algebra in the web browser itself (that’s called “edge computing” for buzzword-loving people).
 Thank you Alastair. So, hi everybody, welcome to Foss4G one more time. I am Ivan Sanchez for those who don't know me. I am an Erd, I dropped off from computer science which means I don't have any formal education in topography, photogrammetry, geodesy or anything of the like. I did buy GPS back in 2006 and since I did that it has been a downward spiral to all the things that we longed for. So, before I get into the meat of this, I want to say a few things. I don't like Zoom. It has had a few complaints over the last few weeks regarding security and its privacy and it's not flossing and because we are part of FOSSGO and the OS in the geostransfer open source, I do think that it is our moral duty to take a stand for open source. So, you know, try it, you'd see, it's fine and it works. I know that there are practical concerns here and I know that the organizers have not let in perfect media of good. It's still, I shouldn't be prevented from saying, hey, open source rules. Also, since we are in the UK now, I am going to speak chiefly English. So in every point that I could, I am writing color as it should. Also, since we are going to talk about multi-spectral, I want to remember that usually we only care about the visible color bands of the spectrum, which is the red, green and blue and then we build up everything from there. But multi-spectral is about, say, worrying about the whole spectrum, the whole spectrum. If you don't get this joke, it's because you are not British. I mean, a British joke, we cannot go home now, or wait for a while. Fine. So, this is not really about WebGL, this is about Big Event Tech for a while, which is going to be open GL and WebGL. It's really a bit the same until, unless you get to specifics. So, something that I like from both GIS and video games is that they are multi-sip, you can get into a very specific field within GIS or within video games and stay there for 20 years of your professional life. There's a lot to unpack. I just love that fact. You can go nerd into one very specific field and just nerd out for years. Both fields have acronyms that don't make any sense whatsoever. And I guess that a lot of people here will know what EPSG stands for, of the top of your heads. I should give a price to the first one in chat who says that. But if you're new to GIS, you might not be aware that EPSG stands for European Petroleum Survey Group, which makes no sense because an EPSG code should be called a Datum Projection Code. That name makes much more sense. On the other hand, in video game tech and open GL, we have something called VBO, which stands for Berthold's Buffer Object. The problem with that is that Berthold's Buffer Object does not hold vertices, it holds attributes for vertices. It's not a buffer because it's not a temporary data structure. It's a semi-permanent data structure. And it's not an object because it's not object-oriented. So it really should be called something like actually you'd write only memory allocation. That would make much more sense. So both fields have weird, weird coordinates. When you go into GIS first, it's like, yeah, everything is like long and then you start reading. But all these crazy, really, really scary words that I'm afraid of. If you go into video games, you learn that there's like five or six different coordinate systems, depending on what you're looking at. Most of them are kind of understandable, especially the world coordinates are kind of easy to understand. You plant all your game map and everything has a coordinate relative to the game map. The screen coordinates are also easy to understand. There's a problem where the screen coordinates start from one corner of the screen and then all the mathematical coordinates start at the bottom of the screen because one is cartesian and that's relative to where the electronics start indexing things. So it's really weird. Then depending on technology, the textures can be rough or not rough. So the coordinates can be negative and still the stuff will work. It's messy. The most difficult one to understand are the clip space coordinates. To think about to get an idea of how this works, you have to think that the line from your eye forward, that's the set axis. So everything up will be the x, everything to the left or right is y and everything deep is that. So whenever you're doing video games, it's doing a coordinate projection from the world coordinates to this clip space coordinates. And it flattens everything onto your screen, making another coordinate projection to scale things up and down. Both fields have more than RGB truer pixels. In GAS, it's useful to work with infrared and extra-hydro productivity and panchromatic stuff and three or four different kinds of elevation, which is always fun, as most people will know. In video games, do you also have more stuff than RGB? You have specifically important hard bump maps, which are normal maps. And then you have how much amount of light reflects on a surface, which is the peculiar map. And then if something is moving, you can get a motion map to make something more or less blurry. And you can get something like in the order of 20 different parameters for one pixel. In both cases, and this is quite important thing to realize, the urgent color that you're looking at in your screen is a function of the data for each pixel. Usually, if we have a true color image, we just project, we just map that one to one to the screen color. In video games, it's usually the texture color times the light, times the reflectivity, but the light has to be cross-multiplied with vector multiplication with the first light direction times the color of light. But the direction of light is modified with simple sum by the bump map. So something that is rough will reflect light from different sources and it creates a weird effect. And this is, if you are going to take one thing away from the stock, it has to be this. Video games are doing raster algebra. So something like this that I'm going to put into the chat right now. This is doing raster algebra. This is just the texture with light source. And if you can read this code, which I can, it's getting some positions of, it's getting the position of the current pixel here. And then it's extrapolating the position of a different pixel in the direction towards the light or away from the light. I'm not sure about this point. Then it's fetching the color of the current pixel and the pixel near the light comparing their luminosity. And then it's brightening up or brightening down the pixel in relation to the luminosity of a pixel nearer or farther to light. This is raster algebra. And this is happening 20 times per second in my machine. This is the one thing I want you all to remember. This is doing raster algebra. So the big difference between GIS and games is that GIS is geared towards accuracy and games must be fast. This is an actual quote from people who are worth talking about open stream. Something like 10 or 15 years ago. Open stream has an accuracy limit of half a centimeter more or less. And people with cadastral applications, they were just burning that saying that if this cannot hold values less than one half a centimeter worldwide, this is completely useless. That hasn't been the case for open stream. But games are a completely different thing. They have to work fast. If something doesn't work fast, a full resolution, you're going to get a crazy gamer with very expensive equipment complaining on YouTube that oh my God, this video game doesn't run on full resolution with 4K shadows and multi-volumetric things all around. So this has been historically a very different set of things to care about. However, we are, I think we are converging a bit about that. GIS has been faster because of tiles a lot. Since 2005-ish, tiles have been speeding things up immensely because you only worry about the amount of data that you need to worry about. Before that time, the early 2000s, the GIS field was all about having all the data at all the time in full resolution. When you realize that you only have to show partial data to cover your stream, tiles became much easier. When you realize that you can cache things on web servers, then tiles were the way forward. We have had that for a long time with WMTS. Vector files have taken over vector data for less than five years or so. Then Cloud Optimized Geotips, they are really a tile parameter in one file that you can query by range. It's the same way. You just care about tiles and transfer small amounts of data up and down to cover your needs, no more. Now GPUs have been getting more and more fast and also more accurate over the years because whatever reason you want to say, the point being, all GPUs back in the late 19s and early 2000s, they could only worry about a few triangles and 8-bit textures. Internally, they moved forward to 24-bit floating point numbers for all the values internally in the shader. Then a few years later, you can just push 32-bit float pointing numbers in data queues, like actual data queues with three dimensions to query the data. Unfortunately for us, this tech is everywhere. We can get 98% of support in web browsers for WebGL and something about 75 for WebGL2, which is not too much until Apple gets into the one background, but it's high enough to consider it a wide support. Now the sweet spot for this, and this is the thing I am convinced it can be done and should be done, is that you can get a raster tile with either 8-bit color, like visible red and blue, or 16-bit per pixel or 32-bit per pixel per sample because let's remember, we have several bands on every image. We can put that into a WebGL texture or an OpenGL texture directly with no transformation whatsoever. We can just dump data into video memory and let the GPU convert all those textures with a lot of data into a visible RGBA color. You can do real-time raster algebra because remember, and I am going to say this once again, you can fight all the times you want with me about this fact, video games have been doing raster algebra better and 100 times faster than any GIS software for years. That's true. Let's complain about this in the coffee room. This is not without challenges because GIS has a lot of history. We like to have our own tools in the CPU. The state of the art, as far as I'm aware, and please correct me about this later, is that every different tool that does raster algebra uses its own specific language. I haven't seen what we computer scientists call a formal, Dan, I forgot about the name, a formal grammar for the raster algebra languages. All the tools have a fairly simple and fairly easy to understand language with just the raster name, the band names, and mathematical operators, but I haven't seen or I haven't noticed a formal specification of those languages. As far as I'm aware, these are different languages which are not going to be compatible sooner or later. I don't think you can copy paste one of these languages to any other tool. We have seen the emergence of raster as a service. This has the downside, I think, of being kind of an evil incentive because what you want, if you are operating raster as a service, what you want is more traffic so you can invoice more money. So you don't really have an incentive for generating less traffic, whereas users do have an incentive for receiving less traffic because it will make things faster. So it's a bit perverse. I don't really know how to handle that, but I think it's there. And also we have metadata. Emily was talking about this briefly in her talk in the previous session. And the thing is that things like the photogrammetry interpretation, it's in the geotifs, but it's hard to read because it's not something that fits into the whole GPU video game graphics process in pipeline. Something like the photogrammetry interpretation is kind of what raster algebra are you going to do to this data, which is the stuff that I'm interested in. So I don't want to choose from five different options. I want to define my own option on how to convert this data to RGB. A lot of the metadata, such as the luminance values or the aperture of the sensors, which are including geotifs from sentient and so on. And there is no straightforward way of translating that into something that we can use for shading it. On the other hand, we have a lot of tech, which is deeply entrenched. We have concepts such as models and sentries and materials in GL. So everything that is about GL, you assume that it's a 3D world. I don't care about 3D world. There is no way to debug this. It's horrible to work with this. There is magic. If you're a comp-c and having to work with the C, this has the low level pointer problems, which you cannot debug, which makes it double horrible. And looking at the specs, it just no. One of my worst nightmares was reading the 200 WMS specification years ago. I don't want to read more 300 specs in PDF that are slightly inconsistent with themselves. So I'm going to share with you the secret of hacking webmats. You do analysis, do morphoses, do synthesis. If you want to do anything with your science and programming, you have to do this. Since these are fancy words, I'm going to translate to English, which is you break it up, you change a part, and you put it back together. Or you are aware of all the moving parts in your system to change one very specific moving part, and you make sure that everything stays together. This is how web mapping library works. You pan the map, get a tile, cut to the URL, fetch it from the network, when it's ready, you just put it in the DOM or render it somehow. What I want to do here is put something in the middle in a surgical way. You do the request for the tile, you get the tile data, you put that tile data into the GPU, you run the thing, and when you have the RGBA result, you put that back into the map as if it were a tile. Because the format of the RGBA result of the raster algebra is completely compatible with tile. Also, if you go into the graphics processing pipeline, usually you are talking about a 3D model with triangles and the camera perspective transformations and so on. I don't care about that. What I want is just have one square, do all the raster algebra in the pixels, get the RGB tile. What's available right now? They are proof of concept. You can search for this project, tile.gl, and there's a branch on the open layers project that I was working in. Webgl1, which is more ubiquitous, it has some restrictions on how much data you can load safely, which can impact some applications. Webgl2 does not have that application, and you can actually have an actual data cube of at least 2000 pixels each side with 32-bit floating point data natively with little precision loss, which is absolutely fantastic. You can just plug GOTFJS to read GOTF data and dump the data, and it just works. Bite packing to support 16-byte data in Webgl1 is possible. It's messy, though. There are a few issues that I was hitting just today when I was preparing the demos at the bounding boxes of GOTFs. They are relative to the bounding box of GOTF itself. If you have a GOTF, you just slice that in four and each of those in four and so on and so on. You don't care about global tile grids. You don't care about any kind of GOTF grids. They don't really match. When I'm requesting a tile from GOTFJS, it's doing resampling. That is doing in the CPU right now, and it's becoming a problem because it takes too much memory. It runs in the CPU hot. It's not perfect. One good thing, okay, we cannot assume that the tiles are the same. If we go back to this thing, we don't do the URLs for tiles. We query the GOTF, and then we extract the actual GOTF tile, not the globally referenced tile, and we work from there. It's possible. You have to redo all the logic, and it's awful. You can also just skip all that, go Webgl inside GOTF and do all the resampling in GPU. That means that you should have shaders in the GOTF logic and do something and for the program interpretations. It's time for demos, and I have less than one minute for demos, but during the questions, I'm going to ask people to tell me how to interpret images. Let's just hope this goes well. First thing I want to do here is I want to pull this one here. This is an indexed GOTF from Vincent that prompted me to do this a few months ago. This is loading a cloud optimized GOTF, which is 3 gigabytes in size. It's reading the indexed value of that GOTF, and it's applying one color depending on the category. The cool thing is I just can change the way I'm visualizing this data in real time or near real time. I think this is just so cool. There's a bit of a lag, which is not really going to be in the video, I guess. That's because of the way that OpenNAS is post-processing for the projected tiles. The pipeline is not too trivial, but it works, and I'm really happy that it works. I'm hoping to put this online in a short-term. The second thing is this demo for fully multispectral support in Live.gl2. What I'm doing here is I'm loading four different cloud optimized GOTF tiles, and I'm fetching data from all of them and storing them in 15 different variables here, and I can just do raster algebra here and output a color. If I want to show true color, what I would do instead is for the red, I want the true color red, for the green, I want the true color green, and for the blue, I want the true color blue. That would show me true color. If I want to replace, let's say, if I want to do false color, I will just put near-introlet, which is the band 8, I think. Since the band 8 is, this is the raw data from the GOTF, the values from true color are from 0 to 1, but the values from all the other bands are the raw values from 0 to 32,000, because it's a 15 significant bit value, so I can just divide this number, and hey, I have false color for all of a sudden. So the demos are kind of working, and I think it's going to be time for questions. Or please, if you don't have any question, tell me a way to interpret 12 bands of sentinel data at the same time. What kind of stuff can I do with this that makes sense and doesn't look horribly ideal? You'll be glad to know that we do have a few questions. So first question here is from Tom Chadwin, and it says, in layperson's terms, what else other than handling much larger data does WebGL to offer over WebGL? It offers a lot of things that we don't care about. That would be the answer. I don't care about transformation buffers. I don't care about being able to query textures in the vertex shader. I don't care about that. So we can go back to here. WebGL 2, you have a lot of stuff all over the graphics pipeline that are not really useful for Rust. So we have a question from Brett Karloch, and I'll try and get this right. So the context is a slide that you were showing about the flow of tiling into a GPU and then out to web maps. And the question was, so are the last two steps on that slide analogous? What would those last two steps be analogous to for gaming engines? You don't have those steps in gaming engines. That's a thing. That's a thing. You have to be aware of all the moving pieces and mix and match the pieces you want. Because in gaming, we are used to having one output frame buffer. Just screen is the final result. That's it. That's what we're used to. When I'm doing this for web maps, the output frame buffer is not the final step. I want to pipe that data down somewhere else. There is something that you can think about which is rendering frame buffers and frame buffers. For this, you have to look at all this. The example of this is the monitors in Nuke and 3D. Some video games allow you to have video screens inside the video game itself. So what's that doing is rendering a scene and then using that scene as a texture in the video game itself. That's something that can be done. I'm trying to find a quick example of that. I'm not doing that. But my point is, when you're dealing with this, when you're dealing with 3D frameworks, I think it's a bad idea to think that the framework is all you can do. When you look at all the pieces that are moving together and you can mix and max those pieces together, that's when you discover all the possibilities that are on them. Cool. I had a quick question for you. It's more to do with the fact that something you said right at the beginning about how they're very different but very similar types of technology. How would you see any of these technologies working? On Geospatial, generally, we're trying to look at a scientific output and in video gaming, I guess we're trying to look at an artistic output. How do you see those two things rubbing up against each other? One of the things that this thing fits with Tom's question, one of the things that you can do with WebGL 2 is instead of having RGB as an output, you can have a buffer as an option, as an output. I mean, so instead of having 8-bit RGB on your screen and then that into a map tile and that into open layers or leaflet, you can have a buffer of 32-bit floating point data and dump that to a GeoTiff. You can do that. Okay, cool. That's absolutely brilliant. I'm afraid we've run out of questions and run out of time as well. Nobody's going to tell me how to do something fancy with this? Oh, come on. I wanted to really play with Banshee because it's fun and weird.
 Thank you Alastair. So, hi everybody, welcome to Foss4G one more time. I am Ivan Sanchez  for those who don't know me. I am an Erd, I dropped off from computer science which  means I don't have any formal education in topography, photogrammetry, geodesy or anything  of the like. I did buy GPS back in 2006 and since I did that it has been a downward spiral  to all the things that we longed for. So, before I get into the meat of this, I want  to say a few things. I don't like Zoom. It has had a few complaints over the last few  weeks regarding security and its privacy and it's not flossing and because we are part  of FOSSGO and the OS in the geostransfer open source, I do think that it is our moral  duty to take a stand for open source. So, you know, try it, you'd see, it's fine and it  works. I know that there are practical concerns here and I know that the organizers have not  let in perfect media of good. It's still, I shouldn't be prevented from saying, hey, open  source rules. Also, since we are in the UK now, I am going to speak chiefly English.  So in every point that I could, I am writing color as it should. Also, since we are going  to talk about multi-spectral, I want to remember that usually we only care about the visible  color bands of the spectrum, which is the red, green and blue and then we build up everything  from there. But multi-spectral is about, say, worrying about the whole spectrum, the whole  spectrum. If you don't get this joke, it's because you are not British. I mean, a British  joke, we cannot go home now, or wait for a while. Fine. So, this is not really about  WebGL, this is about Big Event Tech for a while, which is going to be open GL and WebGL.  It's really a bit the same until, unless you get to specifics. So, something that I like  from both GIS and video games is that they are multi-sip, you can get into a very specific  field within GIS or within video games and stay there for 20 years of your professional  life. There's a lot to unpack. I just love that fact. You can go nerd into one very specific  field and just nerd out for years. Both fields have acronyms that don't make any sense whatsoever.  And I guess that a lot of people here will know what EPSG stands for, of the top of your  heads. I should give a price to the first one in chat who says that. But if you're  new to GIS, you might not be aware that EPSG stands for European Petroleum Survey Group,  which makes no sense because an EPSG code should be called a Datum Projection Code.  That name makes much more sense. On the other hand, in video game tech and open GL, we have  something called VBO, which stands for Berthold's Buffer Object. The problem with that is that  Berthold's Buffer Object does not hold vertices, it holds attributes for vertices. It's not  a buffer because it's not a temporary data structure. It's a semi-permanent data structure.  And it's not an object because it's not object-oriented. So it really should be called something like  actually you'd write only memory allocation. That would make much more sense.  So both fields have weird, weird coordinates. When you go into GIS first, it's like, yeah,  everything is like long and then you start reading. But all these crazy, really, really  scary words that I'm afraid of. If you go into video games, you learn that there's like  five or six different coordinate systems, depending on what you're looking at. Most  of them are kind of understandable, especially the world coordinates are kind of easy to  understand. You plant all your game map and everything has a coordinate relative to the  game map. The screen coordinates are also easy to understand. There's a problem where  the screen coordinates start from one corner of the screen and then all the mathematical  coordinates start at the bottom of the screen because one is cartesian and that's relative  to where the electronics start indexing things. So it's really weird. Then depending on technology,  the textures can be rough or not rough. So the coordinates can be negative and still  the stuff will work. It's messy. The most difficult one to understand are the clip space coordinates.  To think about to get an idea of how this works, you have to think that the line from  your eye forward, that's the set axis. So everything up will be the x, everything to  the left or right is y and everything deep is that. So whenever you're doing video games,  it's doing a coordinate projection from the world coordinates to this clip space coordinates.  And it flattens everything onto your screen, making another coordinate projection to scale  things up and down. Both fields have more than RGB truer pixels. In GAS, it's useful  to work with infrared and extra-hydro productivity and panchromatic stuff and three or four different  kinds of elevation, which is always fun, as most people will know. In video games, do  you also have more stuff than RGB? You have specifically important hard bump maps, which  are normal maps. And then you have how much amount of light reflects on a surface, which  is the peculiar map. And then if something is moving, you can get a motion map to make  something more or less blurry. And you can get something like in the order of 20 different  parameters for one pixel. In both cases, and this is quite important thing to realize,  the urgent color that you're looking at in your screen is a function of the data for  each pixel. Usually, if we have a true color image, we just project, we just map that one  to one to the screen color. In video games, it's usually the texture color times the light,  times the reflectivity, but the light has to be cross-multiplied with vector multiplication  with the first light direction times the color of light. But the direction of light is modified  with simple sum by the bump map. So something that is rough will reflect light from different  sources and it creates a weird effect. And this is, if you are going to take one thing  away from the stock, it has to be this. Video games are doing raster algebra. So something  like this that I'm going to put into the chat right now. This is doing raster algebra. This  is just the texture with light source. And if you can read this code, which I can, it's  getting some positions of, it's getting the position of the current pixel here. And then  it's extrapolating the position of a different pixel in the direction towards the light or  away from the light. I'm not sure about this point. Then it's fetching the color of the  current pixel and the pixel near the light comparing their luminosity. And then it's  brightening up or brightening down the pixel in relation to the luminosity of a pixel nearer  or farther to light. This is raster algebra. And this is happening 20 times per second  in my machine. This is the one thing I want you all to remember. This is doing raster  algebra. So the big difference between GIS and games is that GIS is geared towards accuracy  and games must be fast. This is an actual quote from people who are worth talking about  open stream. Something like 10 or 15 years ago. Open stream has an accuracy limit of  half a centimeter more or less. And people with cadastral applications, they were just  burning that saying that if this cannot hold values less than one half a centimeter worldwide,  this is completely useless. That hasn't been the case for open stream. But games are a completely  different thing. They have to work fast. If something doesn't work fast, a full resolution,  you're going to get a crazy gamer with very expensive equipment complaining on YouTube  that oh my God, this video game doesn't run on full resolution with 4K shadows and multi-volumetric  things all around. So this has been historically a very different set of things to care about.  However, we are, I think we are converging a bit about that. GIS has been faster because  of tiles a lot. Since 2005-ish, tiles have been speeding things up immensely because  you only worry about the amount of data that you need to worry about. Before that time,  the early 2000s, the GIS field was all about having all the data at all the time in full  resolution. When you realize that you only have to show partial data to cover your stream,  tiles became much easier. When you realize that you can cache things on web servers,  then tiles were the way forward. We have had that for a long time with WMTS. Vector files  have taken over vector data for less than five years or so. Then Cloud Optimized Geotips,  they are really a tile parameter in one file that you can query by range. It's the same  way. You just care about tiles and transfer small amounts of data up and down to cover  your needs, no more. Now GPUs have been getting more and more fast and also more accurate  over the years because whatever reason you want to say, the point being, all GPUs back  in the late 19s and early 2000s, they could only worry about a few triangles and 8-bit  textures. Internally, they moved forward to 24-bit floating point numbers for all the  values internally in the shader. Then a few years later, you can just push 32-bit float  pointing numbers in data queues, like actual data queues with three dimensions to query  the data. Unfortunately for us, this tech is everywhere. We can get 98% of support in  web browsers for WebGL and something about 75 for WebGL2, which is not too much until  Apple gets into the one background, but it's high enough to consider it a wide support.  Now the sweet spot for this, and this is the thing I am convinced it can be done and should  be done, is that you can get a raster tile with either 8-bit color, like visible red  and blue, or 16-bit per pixel or 32-bit per pixel per sample because let's remember, we  have several bands on every image. We can put that into a WebGL texture or an OpenGL  texture directly with no transformation whatsoever. We can just dump data into video memory and  let the GPU convert all those textures with a lot of data into a visible RGBA color. You  can do real-time raster algebra because remember, and I am going to say this once again, you  can fight all the times you want with me about this fact, video games have been doing raster  algebra better and 100 times faster than any GIS software for years. That's true. Let's  complain about this in the coffee room. This is not without challenges because GIS has  a lot of history. We like to have our own tools in the CPU. The state of the art, as  far as I'm aware, and please correct me about this later, is that every different tool that  does raster algebra uses its own specific language. I haven't seen what we computer  scientists call a formal, Dan, I forgot about the name, a formal grammar for the raster  algebra languages. All the tools have a fairly simple and fairly easy to understand language  with just the raster name, the band names, and mathematical operators, but I haven't  seen or I haven't noticed a formal specification of those languages. As far as I'm aware, these  are different languages which are not going to be compatible sooner or later. I don't  think you can copy paste one of these languages to any other tool. We have seen the emergence  of raster as a service. This has the downside, I think, of being kind of an evil incentive  because what you want, if you are operating raster as a service, what you want is more  traffic so you can invoice more money. So you don't really have an incentive for generating  less traffic, whereas users do have an incentive for receiving less traffic because it will  make things faster. So it's a bit perverse. I don't really know how to handle that, but  I think it's there. And also we have metadata. Emily was talking about this briefly in her  talk in the previous session. And the thing is that things like the photogrammetry interpretation,  it's in the geotifs, but it's hard to read because it's not something that fits into  the whole GPU video game graphics process in pipeline. Something like the photogrammetry  interpretation is kind of what raster algebra are you going to do to this data, which is  the stuff that I'm interested in. So I don't want to choose from five different options.  I want to define my own option on how to convert this data to RGB. A lot of the metadata, such  as the luminance values or the aperture of the sensors, which are including geotifs from  sentient and so on. And there is no straightforward way of translating that into something that  we can use for shading it. On the other hand, we have a lot of tech, which is deeply entrenched.  We have concepts such as models and sentries and materials in GL. So everything that is  about GL, you assume that it's a 3D world. I don't care about 3D world. There is no way  to debug this. It's horrible to work with this. There is magic. If you're a comp-c and  having to work with the C, this has the low level pointer problems, which you cannot debug,  which makes it double horrible. And looking at the specs, it just no. One of my worst  nightmares was reading the 200 WMS specification years ago. I don't want to read more 300  specs in PDF that are slightly inconsistent with themselves. So I'm going to share with  you the secret of hacking webmats. You do analysis, do morphoses, do synthesis. If you  want to do anything with your science and programming, you have to do this. Since these  are fancy words, I'm going to translate to English, which is you break it up, you change  a part, and you put it back together. Or you are aware of all the moving parts in your  system to change one very specific moving part, and you make sure that everything stays  together. This is how web mapping library works. You pan the map, get a tile, cut to  the URL, fetch it from the network, when it's ready, you just put it in the DOM or render  it somehow. What I want to do here is put something in the middle in a surgical way.  You do the request for the tile, you get the tile data, you put that tile data into the  GPU, you run the thing, and when you have the RGBA result, you put that back into the  map as if it were a tile. Because the format of the RGBA result of the raster algebra is  completely compatible with tile. Also, if you go into the graphics processing pipeline,  usually you are talking about a 3D model with triangles and the camera perspective transformations  and so on. I don't care about that. What I want is just have one square, do all the  raster algebra in the pixels, get the RGB tile. What's available right now? They are  proof of concept. You can search for this project, tile.gl, and there's a branch on  the open layers project that I was working in. Webgl1, which is more ubiquitous, it has  some restrictions on how much data you can load safely, which can impact some applications.  Webgl2 does not have that application, and you can actually have an actual data cube  of at least 2000 pixels each side with 32-bit floating point data natively with little precision  loss, which is absolutely fantastic. You can just plug GOTFJS to read GOTF data and  dump the data, and it just works. Bite packing to support 16-byte data in Webgl1 is possible.  It's messy, though. There are a few issues that I was hitting just  today when I was preparing the demos at the bounding boxes of GOTFs. They are relative  to the bounding box of GOTF itself. If you have a GOTF, you just slice that in four and  each of those in four and so on and so on. You don't care about global tile grids. You  don't care about any kind of GOTF grids. They don't really match. When I'm requesting a  tile from GOTFJS, it's doing resampling. That is doing in the CPU right now, and it's  becoming a problem because it takes too much memory. It runs in the CPU hot. It's not perfect.  One good thing, okay, we cannot assume that the tiles are the same. If we go back to this  thing, we don't do the URLs for tiles. We query the GOTF, and then we extract the actual  GOTF tile, not the globally referenced tile, and we work from there. It's possible. You  have to redo all the logic, and it's awful. You can also just skip all that, go Webgl inside  GOTF and do all the resampling in GPU. That means that you should have shaders in the  GOTF logic and do something and for the program interpretations. It's time for demos, and  I have less than one minute for demos, but during the questions, I'm going to ask people  to tell me how to interpret images. Let's just hope this goes well. First thing I want  to do here is I want to pull this one here. This is an indexed GOTF from Vincent that  prompted me to do this a few months ago. This is loading a cloud optimized GOTF, which is  3 gigabytes in size. It's reading the indexed value of that GOTF, and it's applying one  color depending on the category. The cool thing is I just can change the way I'm visualizing  this data in real time or near real time. I think this is just so cool. There's a bit  of a lag, which is not really going to be in the video, I guess. That's because of the  way that OpenNAS is post-processing for the projected tiles. The pipeline is not too trivial,  but it works, and I'm really happy that it works. I'm hoping to put this online in a  short-term. The second thing is this demo for fully multispectral support in Live.gl2.  What I'm doing here is I'm loading four different cloud optimized GOTF tiles, and I'm fetching  data from all of them and storing them in 15 different variables here, and I can just  do raster algebra here and output a color. If I want to show true color, what I would  do instead is for the red, I want the true color red, for the green, I want the true  color green, and for the blue, I want the true color blue. That would show me true color.  If I want to replace, let's say, if I want to do false color, I will just put near-introlet,  which is the band 8, I think. Since the band 8 is, this is the raw data from the GOTF,  the values from true color are from 0 to 1, but the values from all the other bands are  the raw values from 0 to 32,000, because it's a 15 significant bit value, so I can just  divide this number, and hey, I have false color for all of a sudden. So the demos are  kind of working, and I think it's going to be time for questions. Or please, if you  don't have any question, tell me a way to interpret 12 bands of sentinel data at the  same time. What kind of stuff can I do with this that makes sense and doesn't look horribly  ideal? You'll be glad to know that we do have a few questions. So first question here is  from Tom Chadwin, and it says, in layperson's terms, what else other than handling much  larger data does WebGL to offer over WebGL? It offers a lot of things that we don't care  about. That would be the answer. I don't care about transformation buffers. I don't care  about being able to query textures in the vertex shader. I don't care about that. So  we can go back to here. WebGL 2, you have a lot of stuff all over the graphics pipeline  that are not really useful for Rust. So we have a question from Brett Karloch, and I'll  try and get this right. So the context is a slide that you were showing about the flow  of tiling into a GPU and then out to web maps. And the question was, so are the last two  steps on that slide analogous? What would those last two steps be analogous to for gaming  engines? You don't have those steps in gaming engines. That's a thing. That's a thing. You  have to be aware of all the moving pieces and mix and match the pieces you want. Because  in gaming, we are used to having one output frame buffer. Just screen is the final result.  That's it. That's what we're used to. When I'm doing this for web maps, the output frame  buffer is not the final step. I want to pipe that data down somewhere else. There is something  that you can think about which is rendering frame buffers and frame buffers. For this,  you have to look at all this. The example of this is the monitors in Nuke and 3D. Some  video games allow you to have video screens inside the video game itself. So what's that  doing is rendering a scene and then using that scene as a texture in the video game  itself. That's something that can be done. I'm trying to find a quick example of that.  I'm not doing that. But my point is, when you're dealing with this, when you're dealing  with 3D frameworks, I think it's a bad idea to think that the framework is all you can  do. When you look at all the pieces that are moving together and you can mix and max those  pieces together, that's when you discover all the possibilities that are on them.  Cool. I had a quick question for you. It's more to do with the fact that something you  said right at the beginning about how they're very different but very similar types of technology.  How would you see any of these technologies working? On Geospatial, generally, we're trying  to look at a scientific output and in video gaming, I guess we're trying to look at an  artistic output. How do you see those two things rubbing up against each other?  One of the things that this thing fits with Tom's question, one of the things that you  can do with WebGL 2 is instead of having RGB as an output, you can have a buffer as an  option, as an output. I mean, so instead of having 8-bit RGB on your screen and then that  into a map tile and that into open layers or leaflet, you can have a buffer of 32-bit  floating point data and dump that to a GeoTiff. You can do that.  Okay, cool. That's absolutely brilliant. I'm afraid we've run out of questions and run  out of time as well.  Nobody's going to tell me how to do something fancy with this? Oh, come on. I wanted to  really play with Banshee because it's fun and weird.
FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_0_359.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_360_407.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_408_707.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_708_803.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_804_971.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_972_1871.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_1872_2147.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_2148_2207.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_2208_3347.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_3348_3707.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_3708_9059.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_9060_10823.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_10824_12143.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_12144_12515.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_12516_12947.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_12948_13151.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_13152_13571.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_13572_24551.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_24552_25259.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_25260_25895.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_25896_26615.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_26616_28067.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_28068_28163.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_28164_28331.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_28332_28667.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_28668_28763.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_28764_28835.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_28836_29471.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_29472_30131.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_30132_30299.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_30300_30407.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_30408_30455.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_30456_34895.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_34896_35675.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_35676_36023.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_36024_39155.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_39156_40571.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_40572_40943.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_40944_40991.png FOSS4GUKOnline_2020_Sentinel_Sanchez_Ortega_xEzcf5yd9X0_40992_41038.png