OnEarth: NASA's Boundless Solution to Rapidly Serving Geographic Imagery
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Roberts, Joe
OnEarth is an open source software package that efficiently serves georeferenced raster imagery with virtually zero latency, independent of image size or spatial resolution. The key to OnEarth's speed lies in the use of a unique, multi-resolution file format (Meta Raster Format, or MRF) combined with supporting open source software packages such as the Geospatial Data Abstraction Library (GDAL) and Apache to serve out images via web service protocols such as Web Map Tile Service (WMTS) and Tiled Web Map Service (TWMS), or visualization formats such as Keyhole Markup Language (KML). The emphasis on performance and scalability were strong drivers for developing this specialized package versus using existing software.While OnEarth is currently deployed operationally at several institutions, powering applications across the Earth Science and planetary spectrum, its active development is managed by NASA's Global Imagery Browse Services (GIBS) project. The purpose of GIBS is to provide a complementary historical and near real time (NRT) image archive to NASA's Earth Science data products for a multitude of uses: GIS ingestion, first responder and NRT applications, data search and discovery, decision support, education and outreach.Released as open source to GitHub in October 2013, NASA is encouraging members of the open source community to participate in the evolution of OnEarth—in the roles of developers, evaluators, and users—as a means to vet and enhance its capabilities. This leveraging of efforts not only benefits those who intend to use the software for their own endeavors, it effectively contributes back to NASA by strengthening GIBS and promoting the use and understanding of NASA's vast archive of science imagery and data. Several tools, including the GIBS reference client, Worldview, will be demonstrated as part of this presentation to illustrate the breadth of application and consistent image access speed across-gibs/onearth
 Good afternoon. Thanks for coming. My name is Joe Roberts. I am from the NASA Jet Propulsion Laboratory, just like the presenters before me, or JPL, it's managed by the California Institute of Technology. I work on a team called the Global Imagery Browse Services, or GIBs, it's a collaboration between JPL and also the NASA Goddard Flight Center. So today I'm going to talk about on Earth. So just brief outline, I'm going to explain what on Earth actually is, a little more details about GIBs, get into some more of the technical stuff about on Earth, a little look at performance and metrics, and if there's time maybe do a little preview of some of the client applications that you could build using the on Earth server. And at the end I'm going to just briefly go over our open source project. So what is on Earth? It's basically an open source image processing and server software package. In other words, it's a TAL server. It's intended to provide an out of the box solution for generating geographic imagery. It's a little bit more than just a TAL server, we also include components to build TAL pyramids from Global Mosaic and then store those pyramids in this image archive. So the intent of the archive is to be able to serve TALs to some client application using standard web protocols. For our case we use WMTS, TAL WMS, or KML. TAL WMS is actually an extension of WMS. It was developed at JPL a while back, a long time ago. And it's just a way of creating a predefined set of cache tiles. So when you do a WMS request you already retrieve a set of cache tiles. And I guess the key selling point of on Earth is that it's designed to be lightweight and very performance driven. We use this special file format called the Meadow Raster format or MRF. It's essentially, it's not an image format per se, it's an image container format. And it's, we use this because it has very fast access and we can, we're not limited by image size or resolution. We use a lot of GDAL to generate the MRFs. We have this MRF driver, custom built driver that we use to, you know, run GDAL translate from one image format, JPEG or PNG or whatever, and be able to generate these TAL pyramids into the MRF format. For the actual image server it's essentially just an Apache module. We call it Mod on Earth. And on top of that we have some configuration tools to just kind of help set up the server. Just a little bit background about on Earth. It's been around for a while. It's, its development began originally at JPL in the early 2000s. The lead developer at that time, Luchin, he's now at Esri. But luckily being an open source project we, we still get some pretty good contributions from, from all around. It was formerly known as a TAL WMS server. This is back when, at the time we were just using TAL WMS but now we've expanded to the more commonly used WMTS specification. Some of the other things on Earth has been used for it includes planetarium shows at public museums. It was the first image server to serve out 15 meter global landslide data or imagery. And for, for those of you who remember NASA WorldWin, it was kind of a predecessor to Go-Earth. On Earth was used as the actual image server for, for that application as well. Outside of planetarium I guess there's, we, we have an image server for Mars and the moon and respectively we call that on Mars and on moon. Podac which is the physical oceanography distributed active or archived center is based at JPL. They have this state of the oceans tool to present imagery of other worlds oceans that's also built on this, on their server. Just a couple years ago we started this project Gibbs back in 2011 and on Earth was selected as the primary image server for, for that project as well and on top of Gibbs the client application most commonly used is NASA WorldView. And finally just last year we released a software in open source on GitHub so it's really available and anyone could go up and access and download the software. Okay, so a little bit more about Gibbs. So NASA as you all know has several Earth observation satellites constantly taking measurements of the Earth and we have this really rich and vast image archive. So several useful and assets available to general public scientists and so forth. So the purpose of Gibbs is to provide a full resolution image archive and access services to get to all these different data products. Currently there is about or over 75 global, so that products most of them are available within four hours. It's, it's part of this Earth observation system data or Earth observation system data and information system as this and because it's available to the public we need something that's really scalable, fast, responsive and so forth. So for more, for more information there's a link down there. We also have another presentation, Gibbs related at 4 p.m. that's going to provide some more of the context and historical background of Gibbs. Oops. So I guess one of the key selling points of on Earth is speed. So the key to the speed and storage is this metaraster format that I mentioned earlier. So most of the overhead in actually getting to the imagery or not most but much of the overhead is with, is having to do with the file system. If you have images sitting on, sitting on a machine somewhere, just a bunch of PNGs or millions of tiles just sitting on file folders, that introduces some file system latency. So to reduce that we put everything into one big data file. So, so when you're accessing the data you just have to go to one point without any heavy file system operations. This has been around before. There's, I mean nowadays there's different image file formats or container formats for tile pyramids such as MBTiles and Geo package. So it's been around before that. We didn't really consider anything else during that time. Let's see what else. So with MRF we can support, oops, sorry about that. With MRF we can support multiple projections. For Gibbs we use geographic lat long, Arctic and Antarctic polar stereographic and also Webmer K there for Google maps, Bing maps and so forth. Some of the compression types use. This is for when you pull out the image or you could pull out either JPEG PNGs or even just binary data. I don't know if any of any web clients actually use TIFF but primarily it's going to be JPEG or PNG. I mentioned earlier there's a driver for GDAL that we use to generate the MRFs. And to do that we basically just run GDAL translator. We take an image composite and just convert that to an MRF and on top of that we run some GDAL commands to build up the different zoom levels of that pyramid in order to get the final product. MRF is not itself by itself a single file. It's actually a composed of three different files. The first being the MRF header. It's just an XML based file containing the metadata for the MRF itself. There's also this index file which is basically just a lookup to somewhere in the binary data file which contains all the images. So in the metadata file it's pretty basic actually. It contains just the full base resolution, some of the compression information, the size of the tiles we normally use 512 by 512. There's also additional elements besides this. You can have information about the color maps and other projection information as well. So the data file is pretty simple actually. It's just a bunch of JPEG images or PNGs stitched together side to side starting from the full base resolution at the bottom of the pyramid and moving upward. Each block itself is the actual JPEG image. So once you pull the data out you actually get back that JPEG or PNG. There are some drawbacks to this. Updates are only through a PENs. So there are ways to go back and update some of the data but it's a little more complicated to do that. So the index file is just a binary file containing offset and size. So when you do a lookup for a tile it works really fast. You just look for the roll. So in the top pyramid you have 00, then 01, 02 and so forth. You just go down the roll, look for the offset and then read for that particular size of the tile and you get back that imagery really, really fast. So moving on to the actual, what I call the Honor server. It's just an Apache module. So basically you can run this software with any Apache server. It's just a plug-in that you can configure. It contains the cache metadata. It's basically the information we saw in that MRF header that's stored on the server and we were able to read information about that layer really quick. There's, if for cases where there's no data, so much of the earth if you only have a layer that's just the oceans or just the land you might have some empty data. So we have the special index 00 that just returns the blank empty tile. I mentioned earlier about the common protocols. We have the OGC WMTS, Tiled WMS and KML. If you're familiar with any of those there is get capabilities that a client application can read and figure out what layers are available and so forth. This isn't actually done through the Apache module. We have a separate CGI script that will return you the get capabilities information. We don't exactly follow the specification for WMTS. We have this added time extension. Much of our data has historical imagery so we have to go back in time. So we had this little time snippet if you will to be able to create back for days in the past. With KML, just a little pointer is you always get back a global imagery. So when you pull KML files just for the entire earth whereas with WMTS or Tiled WMS you could pull back the, or you can get the individual tiles. The whole purpose of the module is really just to translate HTTP requests and be able to read MRS and return that to the client application. So a little bit of information about the performance. This is a sample from MODIS data. I guess the key thing to get here is it's possible to get speeds of over 1,000 tiles per second. This is without network latency. So these numbers are a little bit dated. These are from 2011 back when we were doing an evaluation for Gibbs. I'm not sure how this compares with other tile servers nowadays. It would be nice if someone could maybe check it out, give an evaluation and there's better technology out there. I'd love to know about it as well. So, yeah, encourage just some comparison and see what you guys think. And just to see how on earth is working on the wild, these are some sample metrics from Gibbs. There's about 40 million requests two months ago. So there's a pretty decent size of visitors hitting the server. It's not really putting any dent into it. So that's a good thing. Currently, there's over 90 image products. And with Gibbs, we only have 250-meter data. But in the past, we've worked with Landsat, 30-meter, 50-meter data before. So, to help configure this server, we have a couple other tools just to make life a little easier if you're using on earth. So there's this MRF generator. You can use GDaw to basically generate all your MRF files. But this MRF generator abstracts all that GDaw command stuff away from you if you're kind of shy about using that stuff. It also allows for automated processes. We use XML configuration files. So another system could kick down some XML files and be able to generate the imagery for you. So there's also this on earth LRF configurator. We use that to basically generate the server configurations, the metadata for get capabilities and so forth. And a couple other tools, there's this on earth legend generator. So you can take your own color map. We have our own Gibbs format for color maps and just be able to generate a nice little color graphic using that tool. And there's also this metrics tool, which basically it just converts Apache logs into a format that's useful for if you're recording metrics. And this graphic here kind of summarizes how the system works. You prepare the imagery. This assumes you have the images available or global mosaic and you're able to turn that into an image pyramid. And once that occurs, there's some configuration involved with the product layer. And you load that into the server and from that point on, you can serve out the imagery using these common protocols. So for Gibbs, we use on earth in conjunction with the software called tie. It's an image management and workflow system. It helps automate and manage our MRF generation. Another really great thing about it is that it keeps track of the metadata and this is good for data problem non-switches, important for scientists to be able to go back and track where the data has come from. Another great thing about tie is that we can use it for searching and be able to search on the metadata and so forth. This is also in development at JPL. It's not open source, but something good in the amount. So I was going to give a little preview of some of the client applications. It looks like I have some time to maybe check these out. I wasn't sure about the network speeds here. I had a little bit of trouble, but let me see if... So this is worth you. We use this as the Gibbs reference client. And what on earth is really doing is serving the detailed imagery here. If I hit refresh, you can actually see the imagery or the tile is being loaded. But like I mentioned before, I think there are some network speed issues. So I'm not going to do full demo. There is another presentation coming up at 4 p.m. Exposing NASA's Earth observations, I believe it's called. And there's going to be more in-depth talk about worldview. So you can actually see the tile showing up here. That's basically what on earth is doing. There's a couple more other client applications. This one is using Google Earth. You can't really see it on screen here. Just do the screen resolution. But then this is showing sea surface temperature. And there's also a 2D version. This I believe is using leaflet. And showing the same thing, sea surface temperature. There's other satellite layers that you can look up and so forth. One thing I forgot to mention earlier, this on earth server only serves up raster imagery. That doesn't stop you from using a separate server to serve up vector layers. I believe this has some vector information somewhere. We also have this lunar mapping and modeling portal. It's ticking on earth to the moon, I guess. So we have... This is pretty cool. Let's see. Just imagery back to the Apollo era. It's stark or archived imagery of the moon. So it's just pretty cool. So those are just a few couple of sample client applications. Here's some links here. If you want to check it out in your own free time, see what they're all about. So here's a list of other known support clients. Showed off WorldView which uses open layers. There's leaflet, Google Maps, Bing Maps. And of course, this is also available for GIS clients. SRE, the SRE ArcGIS online actually has the gipslers available through their portal. So you could use that for a variety of GIS applications. And of course, there's mobile clients and script load access which is useful for if you want to suck down a bunch of tiles and build a clough free map or something like that. I believe Mapbox did that for their own version of clough free maps. So a little bit about our open source project on GitHub. All of this code is available online. There's a couple other projects as well. There's WorldView and some samples of how you can build your own clients as well. The primary source for the on-earth software is on this NASA Gibbs on-earth site. We're also working to separate the MRF code so we could build that into its own standard. There's some specifications. It's not well documented at this point, but there's some specifications up there to provide you with a little bit more information. Most of the code is in C or C++. Some of the non-performance centric software we use Python just because it's easier to develop and use. And be nice for the open source community to check out the software, provide some feedback whether it's good if you like the performance or if there's some other software that maybe could be a benefit. And also if you participate, there's also the sense of helping out the Earth Science community and aiding, providing some aid to NASA and pushing technology forward. A little bit preview on the future developments. We're working on granular imagery. The challenge is how to be able to serve granularity on our server without sacrificing any performance. There's a couple of items such as multi-dimetry and several improvements to the MRF specifications. We're getting some help from Edzry. They're doing some pretty cool stuff with MRF, making it more cloud-friendly, I guess. And also, oops, I guess this kind of, I don't know what just happened there. So that's basically the end of the slide, anyway. I do want to advertise there's an open position. I don't know too much about it. Yeah, yeah, pretty secretive. But, yeah, that's basically it. I got the email, I'm good. So, I'll leave off with a couple links here. There's just links to our GitHub page. How do you use GIFs? Basically, there's an API. If you want to contact anyone at GIFs, there's the email address in my personal email address as well out there. So, I guess at this point I can take questions. I assume that the serving is all done from those pre-generated tiles. Like, when you talk about the color map, that's not applied on the fly. That's applied during generation of the tiles? Oh, yeah, yeah, that's correct. All the tiles are pre-generated. And during that pre-generation process we pass in the color map and that sticks with the imagery. So, for instance, if you just generate JPEGs, then that's the only thing that server can serve. Is there any interest in, like, doing dynamic conversion? I assume that kind of defeats some of the performance purpose. Yeah, there is interest, but that's a problem as performance. Might be slightly unrelated. Is there a feed of the different data currencies of the different products within the viewer? In other words, how do you best monitor the freshness of the different data that's available? I'm not sure if I followed that question. So, the tiles, the imagery, the tiles, you know, the recency, how do you best evaluate the recency of the different tiles in the imagery? So, for JPEGs anyway, we serve up-daily imagery. So, it's basically just timestamp into that data. There's not really a way to, if you're looking at sub-daily imagery, to see how, there's not a timestamp for how recent that imagery is. Are you guys just showing the EO bands, or is this any other other derived products, like CIS concentration, cloud cover analysis, et cetera? Is that going to go into GIBs, I mean, that sort of thing? Or is that a question for the other topic? Yeah, there is. So, for GIBs there is, if I go to worldview, there are several layers of science products on here. So, in this one, you know, oops, you actually, you guys don't know. Let me try to get here. So, yeah, there's different size parameters that you can get. The list of compatible clients is quite a few. So, yeah, there's a lot of different Mittens and biggestiana rendezvous between**** spilled stits and You mentioned that the WMTS had a little extra thing for time. Is that above and beyond sort of the WMTS standard? And then does the second up question is, does the WMS support the time coordinate or the time parameters? So WMTS is something we added to, it's not in the WMTS spec, it's something we added and I think, I'm not a WMTS expert but I think there is a time extension. And so that just follows along. So your server support that part of the WMS? Right, right, right. Did you put up the other slide that you're going to start with? Oh yeah, yeah. I was just wondering with the MRF format, if imagery is updated four hours from live and the entire format must be appended for updates, how does that happen and how have you dealt with that? Does the entire file need to be regenerated when the data is updated? Yeah, pretty much at this point. About how long does it take to generate the MRF file? That totally depends on the size of the imagery. If it's kilometer size, it's pretty quick. If you're dealing with say three million landslide data, that could take hours. So there is some limitation there if you're dealing with really high resolution imagery is going to take a long time unless you have some really beefy computer power to handle that. I think all the questions are done. Okay. Thank you very much. Thanks.
 Good afternoon. Thanks for coming. My name is Joe Roberts. I am from the NASA Jet Propulsion  Laboratory, just like the presenters before me, or JPL, it's managed by the California  Institute of Technology. I work on a team called the Global Imagery Browse Services,  or GIBs, it's a collaboration between JPL and also the NASA Goddard Flight Center. So  today I'm going to talk about on Earth. So just brief outline, I'm going to explain  what on Earth actually is, a little more details about GIBs, get into some more of the technical  stuff about on Earth, a little look at performance and metrics, and if there's time maybe do  a little preview of some of the client applications that you could build using the on Earth server.  And at the end I'm going to just briefly go over our open source project. So what is  on Earth? It's basically an open source image processing and server software package. In  other words, it's a TAL server. It's intended to provide an out of the box solution for  generating geographic imagery. It's a little bit more than just a TAL server, we also include  components to build TAL pyramids from Global Mosaic and then store those pyramids in this  image archive. So the intent of the archive is to be able to serve TALs to some client  application using standard web protocols. For our case we use WMTS, TAL WMS, or KML.  TAL WMS is actually an extension of WMS. It was developed at JPL a while back, a long  time ago. And it's just a way of creating a predefined set of cache tiles. So when you  do a WMS request you already retrieve a set of cache tiles. And I guess the key selling  point of on Earth is that it's designed to be lightweight and very performance driven.  We use this special file format called the Meadow Raster format or MRF. It's essentially,  it's not an image format per se, it's an image container format. And it's, we use this because  it has very fast access and we can, we're not limited by image size or resolution. We  use a lot of GDAL to generate the MRFs. We have this MRF driver, custom built driver  that we use to, you know, run GDAL translate from one image format, JPEG or PNG or whatever,  and be able to generate these TAL pyramids into the MRF format. For the actual image  server it's essentially just an Apache module. We call it Mod on Earth. And on top of that  we have some configuration tools to just kind of help set up the server. Just a little bit  background about on Earth. It's been around for a while. It's, its development began originally  at JPL in the early 2000s. The lead developer at that time, Luchin, he's now at Esri. But  luckily being an open source project we, we still get some pretty good contributions  from, from all around. It was formerly known as a TAL WMS server. This is back when, at  the time we were just using TAL WMS but now we've expanded to the more commonly used WMTS  specification. Some of the other things on Earth has been used for it includes planetarium  shows at public museums. It was the first image server to serve out 15 meter global  landslide data or imagery. And for, for those of you who remember NASA WorldWin, it was  kind of a predecessor to Go-Earth. On Earth was used as the actual image server for, for  that application as well. Outside of planetarium I guess there's, we, we have an image server  for Mars and the moon and respectively we call that on Mars and on moon. Podac which  is the physical oceanography distributed active or archived center is based at JPL. They  have this state of the oceans tool to present imagery of other worlds oceans that's also  built on this, on their server. Just a couple years ago we started this project Gibbs back  in 2011 and on Earth was selected as the primary image server for, for that project as well  and on top of Gibbs the client application most commonly used is NASA WorldView. And  finally just last year we released a software in open source on GitHub so it's really available  and anyone could go up and access and download the software. Okay, so a little bit more about  Gibbs. So NASA as you all know has several Earth observation satellites constantly taking  measurements of the Earth and we have this really rich and vast image archive. So several  useful and assets available to general public scientists and so forth. So the purpose of  Gibbs is to provide a full resolution image archive and access services to get to all  these different data products. Currently there is about or over 75 global, so that  products most of them are available within four hours. It's, it's part of this Earth  observation system data or Earth observation system data and information system as this  and because it's available to the public we need something that's really scalable, fast,  responsive and so forth. So for more, for more information there's a link down there.  We also have another presentation, Gibbs related at 4 p.m. that's going to provide some more  of the context and historical background of Gibbs. Oops. So I guess one of the key selling  points of on Earth is speed. So the key to the speed and storage is this metaraster  format that I mentioned earlier. So most of the overhead in actually getting to the imagery  or not most but much of the overhead is with, is having to do with the file system. If you  have images sitting on, sitting on a machine somewhere, just a bunch of PNGs or millions  of tiles just sitting on file folders, that introduces some file system latency. So to  reduce that we put everything into one big data file. So, so when you're accessing the  data you just have to go to one point without any heavy file system operations. This has  been around before. There's, I mean nowadays there's different image file formats or container  formats for tile pyramids such as MBTiles and Geo package. So it's been around before  that. We didn't really consider anything else during that time. Let's see what else.  So with MRF we can support, oops, sorry about that. With MRF we can support multiple projections.  For Gibbs we use geographic lat long, Arctic and Antarctic polar stereographic and also  Webmer K there for Google maps, Bing maps and so forth. Some of the compression types  use. This is for when you pull out the image or you could pull out either JPEG PNGs or  even just binary data. I don't know if any of any web clients actually use TIFF but primarily  it's going to be JPEG or PNG. I mentioned earlier there's a driver for GDAL that we  use to generate the MRFs. And to do that we basically just run GDAL translator. We take  an image composite and just convert that to an MRF and on top of that we run some GDAL  commands to build up the different zoom levels of that pyramid in order to get the final  product. MRF is not itself by itself a single file. It's actually a composed of three different  files. The first being the MRF header. It's just an XML based file containing the metadata  for the MRF itself. There's also this index file which is basically just a lookup to somewhere  in the binary data file which contains all the images. So in the metadata file it's pretty  basic actually. It contains just the full base resolution, some of the compression information,  the size of the tiles we normally use 512 by 512. There's also additional elements besides  this. You can have information about the color maps and other projection information as well.  So the data file is pretty simple actually. It's just a bunch of JPEG images or PNGs stitched  together side to side starting from the full base resolution at the bottom of the pyramid  and moving upward. Each block itself is the actual JPEG image. So once you pull the data  out you actually get back that JPEG or PNG. There are some drawbacks to this. Updates  are only through a PENs. So there are ways to go back and update some of the data but  it's a little more complicated to do that. So the index file is just a binary file containing  offset and size. So when you do a lookup for a tile it works really fast. You just look  for the roll. So in the top pyramid you have 00, then 01, 02 and so forth. You just go  down the roll, look for the offset and then read for that particular size of the tile  and you get back that imagery really, really fast. So moving on to the actual, what I call  the Honor server. It's just an Apache module. So basically you can run this software with  any Apache server. It's just a plug-in that you can configure. It contains the cache metadata.  It's basically the information we saw in that MRF header that's stored on the server and  we were able to read information about that layer really quick. There's, if for cases where  there's no data, so much of the earth if you only have a layer that's just the oceans or  just the land you might have some empty data. So we have the special index 00 that just returns  the blank empty tile. I mentioned earlier about the common protocols. We have the OGC  WMTS, Tiled WMS and KML. If you're familiar with any of those there is get capabilities  that a client application can read and figure out what layers are available and so forth.  This isn't actually done through the Apache module. We have a separate CGI script that  will return you the get capabilities information. We don't exactly follow the specification  for WMTS. We have this added time extension. Much of our data has historical imagery so  we have to go back in time. So we had this little time snippet if you will to be able  to create back for days in the past. With KML, just a little pointer is you always get  back a global imagery. So when you pull KML files just for the entire earth whereas with  WMTS or Tiled WMS you could pull back the, or you can get the individual tiles. The whole  purpose of the module is really just to translate HTTP requests and be able to read MRS and  return that to the client application. So a little bit of information about the performance.  This is a sample from MODIS data. I guess the key thing to get here is it's possible  to get speeds of over 1,000 tiles per second. This is without network latency. So these  numbers are a little bit dated. These are from 2011 back when we were doing an evaluation  for Gibbs. I'm not sure how this compares with other tile servers nowadays. It would  be nice if someone could maybe check it out, give an evaluation and there's better technology  out there. I'd love to know about it as well. So, yeah, encourage just some comparison  and see what you guys think. And just to see how on earth is working on the wild, these  are some sample metrics from Gibbs. There's about 40 million requests two months ago.  So there's a pretty decent size of visitors hitting the server. It's not really putting  any dent into it. So that's a good thing. Currently, there's over 90 image products.  And with Gibbs, we only have 250-meter data. But in the past, we've worked with Landsat,  30-meter, 50-meter data before.  So, to help configure this server, we have a couple other tools just to make life a little  easier if you're using on earth. So there's this MRF generator. You can use GDaw to basically  generate all your MRF files. But this MRF generator abstracts all that GDaw command  stuff away from you if you're kind of shy about using that stuff. It also allows for  automated processes. We use XML configuration files. So another system could kick down some  XML files and be able to generate the imagery for you.  So there's also this on earth LRF configurator. We use that to basically generate the server  configurations, the metadata for get capabilities and so forth. And a couple other tools, there's  this on earth legend generator. So you can take your own color map. We have our own Gibbs  format for color maps and just be able to generate a nice little color graphic using  that tool. And there's also this metrics tool, which basically it just converts Apache logs  into a format that's useful for if you're recording metrics. And this graphic here kind  of summarizes how the system works. You prepare the imagery. This assumes you have the images  available or global mosaic and you're able to turn that into an image pyramid. And once  that occurs, there's some configuration involved with the product layer. And you load that  into the server and from that point on, you can serve out the imagery using these common  protocols.  So for Gibbs, we use on earth in conjunction with the software called tie. It's an image  management and workflow system. It helps automate and manage our MRF generation. Another really  great thing about it is that it keeps track of the metadata and this is good for data  problem non-switches, important for scientists to be able to go back and track where the  data has come from. Another great thing about tie is that we can use it for searching and  be able to search on the metadata and so forth. This is also in development at JPL. It's  not open source, but something good in the amount.  So I was going to give a little preview of some of the client applications. It looks  like I have some time to maybe check these out. I wasn't sure about the network speeds  here. I had a little bit of trouble, but let me see if... So this is worth you. We use  this as the Gibbs reference client. And what on earth is really doing is serving the detailed  imagery here. If I hit refresh, you can actually see the imagery or the tile is being loaded.  But like I mentioned before, I think there are some network speed issues. So I'm not  going to do full demo. There is another presentation coming up at 4 p.m. Exposing NASA's Earth  observations, I believe it's called. And there's going to be more in-depth talk about worldview.  So you can actually see the tile showing up here. That's basically what on earth is doing.  There's a couple more other client applications. This one is using Google Earth. You can't  really see it on screen here. Just do the screen resolution. But then this is showing  sea surface temperature. And there's also a 2D version. This I believe is using leaflet.  And showing the same thing, sea surface temperature. There's other satellite layers that you can  look up and so forth. One thing I forgot to mention earlier, this on earth server only  serves up raster imagery. That doesn't stop you from using a separate server to serve  up vector layers. I believe this has some vector information somewhere. We also have  this lunar mapping and modeling portal. It's ticking on earth to the moon, I guess. So  we have... This is pretty cool. Let's see. Just imagery back to the Apollo era. It's  stark or archived imagery of the moon. So it's just pretty cool. So those are just a  few couple of sample client applications. Here's some links here. If you want to check  it out in your own free time, see what they're all about. So here's a list of other known  support clients. Showed off WorldView which uses open layers. There's leaflet, Google  Maps, Bing Maps. And of course, this is also available for GIS clients. SRE, the SRE ArcGIS  online actually has the gipslers available through their portal. So you could use that  for a variety of GIS applications. And of course, there's mobile clients and script  load access which is useful for if you want to suck down a bunch of tiles and build a  clough free map or something like that. I believe Mapbox did that for their own version of  clough free maps. So a little bit about our open source project on GitHub. All of this  code is available online. There's a couple other projects as well. There's WorldView  and some samples of how you can build your own clients as well. The primary source for  the on-earth software is on this NASA Gibbs on-earth site. We're also working to separate  the MRF code so we could build that into its own standard. There's some specifications.  It's not well documented at this point, but there's some specifications up there to provide  you with a little bit more information. Most of the code is in C or C++. Some of the non-performance  centric software we use Python just because it's easier to develop and use. And be nice  for the open source community to check out the software, provide some feedback whether  it's good if you like the performance or if there's some other software that maybe could  be a benefit. And also if you participate, there's also the  sense of helping out the Earth Science community and aiding, providing some aid to NASA and  pushing technology forward. A little bit preview on the future developments. We're working  on granular imagery. The challenge is how to be able to serve granularity on our server  without sacrificing any performance. There's a couple of items such as multi-dimetry and  several improvements to the MRF specifications. We're getting some help from Edzry. They're  doing some pretty cool stuff with MRF, making it more cloud-friendly, I guess. And also,  oops, I guess this kind of, I don't know what just happened there. So that's basically  the end of the slide, anyway. I do want to advertise there's an open position. I don't  know too much about it.  Yeah, yeah, pretty secretive. But, yeah, that's basically it.  I got the email, I'm good.  So, I'll leave off with a couple links here.  There's just links to our GitHub page. How do you use GIFs? Basically, there's an API.  If you want to contact anyone at GIFs, there's the email address in my personal email address  as well out there. So, I guess at this point I can take questions.  I assume that the serving is all done from those pre-generated tiles. Like, when you talk  about the color map, that's not applied on the fly. That's applied during generation  of the tiles?  Oh, yeah, yeah, that's correct. All the tiles are pre-generated. And during that pre-generation  process we pass in the color map and that sticks with the imagery.  So, for instance, if you just generate JPEGs, then that's the only thing that server can  serve. Is there any interest in, like, doing dynamic conversion? I assume that kind of  defeats some of the performance purpose.  Yeah, there is interest, but that's a problem as performance.  Might be slightly unrelated. Is there a feed of the different data currencies of the different  products within the viewer? In other words, how do you best monitor the freshness of the  different data that's available?  I'm not sure if I followed that question.  So, the tiles, the imagery, the tiles, you know, the recency, how do you best evaluate  the recency of the different tiles in the imagery?  So, for JPEGs anyway, we serve up-daily imagery. So, it's basically just timestamp into that  data. There's not really a way to, if you're looking at sub-daily imagery, to see how,  there's not a timestamp for how recent that imagery is.  Are you guys just showing the EO bands, or is this any other other derived products,  like CIS concentration, cloud cover analysis, et cetera? Is that going to go into GIBs,  I mean, that sort of thing? Or is that a question for the other topic?  Yeah, there is. So, for GIBs there is, if I go to worldview, there are several layers  of science products on here. So, in this one, you know, oops, you actually, you guys don't  know. Let me try to get here. So, yeah, there's different size parameters that you can get.  The list of compatible clients is quite a few. So, yeah, there's a lot of different  Mittens and biggestiana rendezvous between**** spilled stits and  You mentioned that the WMTS had a little extra thing for time.  Is that above and beyond sort of the WMTS standard?  And then does the second up question is, does the WMS support the time coordinate or the  time parameters?  So WMTS is something we added to, it's not in the WMTS spec, it's something we added  and I think, I'm not a WMTS expert but I think there is a time extension.  And so that just follows along.  So your server support that part of the WMS?  Right, right, right.  Did you put up the other slide that you're going to start with?  Oh yeah, yeah.  I was just wondering with the MRF format, if imagery is updated four hours from live  and the entire format must be appended for updates, how does that happen and how have  you dealt with that?  Does the entire file need to be regenerated when the data is updated?  Yeah, pretty much at this point.  About how long does it take to generate the MRF file?  That totally depends on the size of the imagery.  If it's kilometer size, it's pretty quick.  If you're dealing with say three million landslide data, that could take hours.  So there is some limitation there if you're dealing with really high resolution imagery  is going to take a long time unless you have some really beefy computer power to handle  that.  I think all the questions are done.  Okay.  Thank you very much.  Thanks.
06_Joe_Roberts_720p_0_5627.png 06_Joe_Roberts_720p_5628_9131.png 06_Joe_Roberts_720p_9132_13823.png 06_Joe_Roberts_720p_13824_14483.png 06_Joe_Roberts_720p_14484_15491.png 06_Joe_Roberts_720p_15492_16307.png 06_Joe_Roberts_720p_16308_19451.png 06_Joe_Roberts_720p_19452_26051.png 06_Joe_Roberts_720p_26052_26159.png 06_Joe_Roberts_720p_26160_26399.png 06_Joe_Roberts_720p_26400_27515.png 06_Joe_Roberts_720p_27516_27875.png 06_Joe_Roberts_720p_27876_28811.png 06_Joe_Roberts_720p_28812_29687.png 06_Joe_Roberts_720p_29688_34511.png 06_Joe_Roberts_720p_34512_34727.png 06_Joe_Roberts_720p_34728_35039.png 06_Joe_Roberts_720p_35040_35447.png 06_Joe_Roberts_720p_35448_36119.png 06_Joe_Roberts_720p_36120_36239.png 06_Joe_Roberts_720p_36240_36635.png 06_Joe_Roberts_720p_36636_36731.png 06_Joe_Roberts_720p_36732_36791.png 06_Joe_Roberts_720p_36792_36851.png 06_Joe_Roberts_720p_36852_38339.png 06_Joe_Roberts_720p_38340_38387.png 06_Joe_Roberts_720p_38388_38639.png 06_Joe_Roberts_720p_38640_39083.png 06_Joe_Roberts_720p_39084_39491.png 06_Joe_Roberts_720p_39492_40739.png 06_Joe_Roberts_720p_40740_41963.png 06_Joe_Roberts_720p_41964_42011.png 06_Joe_Roberts_720p_42012_42179.png 06_Joe_Roberts_720p_42180_42227.png 06_Joe_Roberts_720p_42228_42383.png 06_Joe_Roberts_720p_42384_44123.png 06_Joe_Roberts_720p_44124_44275.png