Open Source as part of an Open Data initiative.
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Wood, Brent
In 2011, the New Zealand Government released the Declaration on Open and Transparent Government, a directive to government agencies to establish open data programmes to enable data reuse, for the wider benefit of New Zealand. This presentation discusses one environmental research agency's approach to providing open data that is not just accessible, but genuinely reusable. Early feedback regarding the programme was positive - but there was a frequent question. Many potential users of open data have had no GIS background, and wanted help in finding tools, and training in the use of those tools to actually make use of the data. In answering this question, it became apparent that a genuinely successful open data programme has three components, with far too many such programmes stopping at the first: Discoverable, accessible metadata and data; Effective, available and affordable tools to work with the data, and; Affordable training for interested users in the use of the tools with the data. This presentation discusses this three pronged approach to an open data programme, actively enhancing and making use of open source and open standards to provide usable open data. One outcome of this programme is presented as “The Democratisation of GIS”, where open Source GIS tools are enabling GIS to become as pervasive as an office suite or web browser on a personal computer. Brent Wood (National Institute of Water and Atrmospheric Research (NIWA) New Zealand)
 People, please have a seat if you want. There are some seats left here. So we're moving to the last presentation on this block on Open Data. It's Brent Wood and he's going, well, as you can see, it is most excellent tourist map. You should have a look at that before you leave here. It is really good. Open Data through Open Standards and Open Source. There you go. And you have to use the microphone for the stream. So otherwise people at home can't hear you. Yep, they might enjoy that. I'm from New Zealand, so I've come just about as far as anyone can come to get here. So yes, greetings from New Zealand and no, that does not have an EPSG code. Start with apology and then ask a favour. And I apologise for that, but I wasn't responsible for the schedule. And my accent, and we tend to talk quickly and people don't understand. If you don't understand, we're both wasting our time and I've come a long way to do this. So please wave, do something if you don't understand me so I can slow down and explain. If you do that, lots of more polite people will be thinking, thank you so much anyway. So the real story. Newa, the National Institute for Water and Water Research, pays my wages. And we study water, atmosphere, climate, fisheries, oceanography, atmospheric science. We're one of a government agencies which operate as a private business. So while we are only shareholder as a government minister, we're required to operate and give that minister a profit to keep his government happy for the financial year. I'm going to discuss the Open Data Initiative in New Zealand. The fact that open data is fundamentally about data reuse. Once you've actually released your data, what happens then? Some of the things we've done to sort of meet those needs and what I see as a more complete solution for an open data package. And most of that's going to be boring text-based stuff and there's some pretty pictures to illustrate things at the end. So lots of text. People who write these things are paid by the syllable. 2011, New Zealand announced a declaration of open transparent government which required – you don't have to read all that, you get the just of it. Core government departments and ministries were directed to make all high-value government data publicly available. Other agencies that weren't core government were encouraged to make their data available. Some of the first outputs came out was NZ Goal. It's a New Zealand government open access licence. It's a creative, common space framework for agencies to release their data under and consistent licensing framework tends to make the legal sides of interoperability flow quite nicely. A very recent enhancement to that was the NZ Goal SE which is a software extension which recognised that code developed by taxpayer funded projects should be made available just as data is. And so we now have an extension to the NZ Goal framework covering code to be released under an open source licence. What was particularly inspiring about that was that the initial draft from the lawyer was put out for public comment. The result of that was putting it on Lumio. All the open source enthusiasts jumped on it in a big way and the resulting document is a genuinely collaborative effort from the public and the open source community into the government process. If you're going to release your data, the whole point of releasing it is enabling people to reuse it. No one's waving yet. You can all understand me okay. One of the problems we went into were there were very few guidelines on how to release your data. We didn't do what the Daniels so sensitively did and set up a nice central organisation that was funded to manage all this. We told a lot of agencies to do it and didn't tell them how. And this was supposed to result in interoperability and commonality across all sectors which was a bit ambitious. Land Information New Zealand had their Spatial Data Infrastructure Cookbook which recommended OGCWFS for Spatial Data. That was about it for government documents describing how agencies should release their data. What Niwa and a lot of other agencies in the environmental space did was we created, we got together, had some discussions and created the Environmental Observation Data Protocol which covers all our sensor-based stuff, all our bio-data-based stuff where people see things, observe things, measure things in the environmental space. And that was across government agencies, the research agencies, regional councils, etc. So it was a very collaborative exercise and it came up with a very much OGC-based approach to every agency making their data available. And the OGC standards are there but we also had things the OGC standards don't cover such as vocabulary servers, small link data stuff, getting into the bio-data with IPT, the GBIF standard which I'll touch on later. So we had some extensions to the core SOS capability. And agencies already had lots of tools and because we settled on industry standards, most of those tools could already do, provide the data using the services that we'd mandated. Lyns introduced last year an institutional GIS for 4,000 people across New Zealand. That's land information New Zealand I mentioned before. The tool they chose is an Australian tool from a company that used to be called Digital Mapping Solutions called Intramaps. Anyone heard of it? One, two. Okay, Intramaps is based on Map Server, QGIS, sits on top of PostGIS. Some open source stack sold and supported as a commercial application. They do a fair amount of QGIS development and other work in the space as well. That's now a nice mix of commercial and FOSS by the major government agency in New Zealand working with spatial data. There's a fairly strong vote of confidence in the open source stack there. So we've got the catalog service, we've got discovery addressed, we've got the other services for delivery. We've made our data available. We can all sit back and go home now and let people do whatever they want. The use of feedback was, what do we do with it? Your usual public user doesn't know about GIS. There was a conference in the US some years ago where the secretary took the notes by hand and all these references to 516 were in her notes. That's your GIS. She'd actually misinterpreted it because you had no idea what GIS was. And most of the people who are using our data don't know what GIS is. So my take on open data, if you're going to make, provide it and facilitate reuse, you need to enable users to do that reuse. And the software tool that does that is called GIS. So we have a program now based on QGIS. We take the free tool, we customise it, we enhance it, we pay the devs to do things with it. And people can now access our data, download a tool and actually use that data. And as part of that process, we're working with DMS again. We're running training sessions around the entire country for people who are interested. It's not free, it's commercial operation. A one-day training course will cost you $200 New Zealand Dollars. That's pretty cheap for a one-day workshop. This is leading to a process I'm describing as democratisation of GIS. And if you've been around as long as me, my first papers were written long-hand on paper. And they went to a typing pool. That typing pool was a team of domain-specific experts providing a service to an institution, which describes your GIS team of yesterday. What Lyns did, there's now 4,000 people with GIS on their desktop. It comes when they start with the organisation. Along with Word, Excel, a PDF reader, you get QGIS. We have open data. We have a tool, and we all know there's lots of people out there wanting to use that data, and we're now enabling to do that. So some of the tools we've put together, IPT, GBIF. Anyone familiar with GBIF? Global Bio Data Information Framework. If someone in New Zealand wants to find out where all the museum specimens or observations of a particular species are, you'd think they'd go to a New Zealand dataset, but you wouldn't. If you were here on a fish called Scaly Gernet, for example, don't ask me why, but there's a museum at the University of Kansas, a long way from New Zealand, a long way from the sea, that has a Scaly Gernet from New Zealand in it. If you want to find all those records, you can't go to an institutional catalogue. You go to a global catalogue that all those institutions upload the data into, and you will find all the records that not just the New Zealand institutions hold, but every institution in the world pretty much. And IPT is the protocol that GBIF provides for uploading data. So you have automatic access to any GBIF provider straight from QGIS. Direct to, it's a river environment classification, 800,000 reaches in New Zealand online as a WMS. Our freshwater fish database plugin, Direct Access, we're working on a SOS plugin for sensor access to data. And we've set up through the New Zealand Open Source Society a geo network instance in the last month. That's being set up to harvest every New Zealand OGC server we can find with a QGIS plugin that will access that server via CSW and allow users to find and store as favourites any web service they can track down. So that's trying to sort of address through the community a sort of consolidated discovery capability. So the complete solution is not just data or access to data, but it's tools to allow you and support you and educate you and train you in how to reuse that data. And as the last very good presentation identified, most of the users were Joe Block's public. And I'm looking at the friends of a local stream who are doing an environmental cleanup of their stream, they're recording water temperatures, they're recording the animals and plants that are living there. That's going into databases that other people can then use and share and learn from. We have sort of botanical societies, we have birdwatches, we have huge amounts of data going in to little NGO community clubs systems that can now be part of a framework and they are enabled by adding all this information to help them direct their efforts. That wasn't supposed to happen. He just took a while, but I don't know why this did that. Anyway, GBIF, through our IBT plugin. This is a pretty picture bit, so a bit more interesting. Look about five minutes to download the data. There's a lot of them. Where you see one point, there's usually lots of them underneath it. But that allows instant access to data that is not normally available within a GIS framework. The IBT protocol is not a GIS, not technically a spatial framework. It uses a Darwin core standard and that data is now available to use within the GIS framework if you use QGIS. Within NIWA, one of the first things we're doing about making data available is making data available to our own staff. We spread across the country, we have different departments. Our left hand doesn't know where there is a right hand, let alone what it's doing. These tools now allow people both in the institution and outside the institution to look at institutional databases and access and visualise and work with the data. So we have 40 years worth of fisheries surveys around the New Zealand coast. You can look at spur dogs, you catch around here as well, they're a global species. So there's one survey or you can look at different species from multiple surveys. Data is available, it's all web servers based. This was not a plugin, we knew our fund was some enhancements, basically allowing you to do things like, which button gives me my light, that one. So nice annotated frameworks, a lot of our stuff's maritime, so we want sort of degree labels because that tends to be on the sea. So we paid the developers to enhance the print composer so you can do nice maps like that. One of the fun things with the polar map, which weighs north. That's using a custom projection because Antarctic Polar Stereographic is actually 180 down. And if I plot that, it looks wrong. It's not, but it looks wrong. So we've got a custom projection with 180 up instead of zero and a nice looking map. The nice thing about this is our specimen database uses an open source tool called specify as the collection management tool. This underlying database is my SQL, non-spatial. Access is enabled through a GDAL virtual data file. You're familiar with those? Essentially it's just a piece of Xml that describes the data source that includes an SQL to run on that data source. When you open it in QGIS, it's as if you've opened the underlying data source, not just the XML file. The XML file is just a pointer to the underlying data. So these people who have a collection tool now can use QGIS to interact directly with that projection tool, generate publication quality graphs directly from it. And we can open it up via a web service and people can actually, the public can actually do this and that will be happening. All of these things are a work in progress. Our SOS client, genuine collaboration. We have a Newell plugin in QGIS. We have a regional council providing a SOS service. We have a base map provided by yet another CRI as a WMS service. And we can plot their various observation stations on the map, click on one, find out what observations are available and display and download those observations. Another work in progress and source polar working on this with us. One of the big issues with SOS is that every SOS server has a slightly different implementation and dialect of SOS. So working through some of those gets interesting. I mentioned before the REC2, the river environment classification, 800,000 freshwater reaches across New Zealand. You can probably see it in the blue lines here. It's a transparent WMS layer so it loads pretty much instantly. It's the back end as opposed to just database. You can open up the tool, click on a reach. In this case it was down at the river mouth there. And click the search upstream button and you'll get a WMS rendition of the upstream network. Which basically means you can stand anywhere and ask if something's here, where could it come from or if it's going, where could it go. This was actually part of another tool that we developed and the interest was such from user feedback that they wanted this as a standalone capability so that was done. Fisheries research, acoustic stock assessment using eco sounders to count fish. Which virtual fish, that's the fish there on that one. We have a commercial database for our imagery that we paid the developer to provide a web API for. And QGIS has an action tool which I'll detail slightly later on. But with the action tool and the web API, the WFS service for all these transit line features includes the image ID in that web accessible database for images as one of the fields. So you can use the image viewer of your choice in a QGIS action, pass it the URL that includes the image ID. You can click on any one of 16,000 transits and see a pretty picture of it. Native QGIS. Now the one that's a bit of fun, I think is fun. It's a good open source story. Back in 2005, version 0.6 of QGIS, we had developed an acoustic database using PostGIS with a colleague of mine Gavin McCauley who I introduced to QGIS. He started using QGIS to look at the acoustic data and wanted the ability to open up an echogram from the map in our echogram analysis software. So he created at that stage the action tool in QGIS. He left Niwa and is currently working for Norway to maintain a Scandinavian connection. And I emailed him to say what we'd just done with the tool that he'd done in the best part of 10 years before. He emailed back a screen cap. He's sitting on a boat north of Norway, due north of Norway. He's running QGIS, connecting the WFS service, running the action tool that he created to open up an echogram for a survey that he actually ran when he was working for Niwa. So New Zealand data opened up in a boat sitting in the North Atlantic by the original developer. I don't think it's a nice circular story. So thank you for your time. That's sort of what we've done and why we've done it and how we're doing it. And I'm happy for questions. So, opportunity for questions. Who can I hand the mic first? Although Aine Beck. Thank you for your talk. Just a question on your opinion on, you mentioned earlier, the initiative for local communities for the river cleanup or whatever that was. How do you see striking that balance between data completeness? So you've got these pretty extensive field and arrays with all the data for potential sort of academic uploads compared to someone who might not be able to create all those fields or submit all that data. So the potential is there for them not to create complete data and yet at the same time you don't want to overwhelm them with forcing them to submit everything. Where do you find that balance? A lot of it is education. This is a slightly interesting example because we are focusing at this stage on just the stream one you mentioned. NEEWA produces what we call a Schmack kit. It's a stream health monitoring kit that we provide through councils to schools, community groups. It has guidelines on what to do, how to do it, the core indicator species to or how to identify them. We're on workshops in that. So any agency that wants to sort of start in that sort of process, we're actually providing tools and training and facilities and through the councils we're trying to establish something that's basically like a toy library. So the kit can be used by one school one day and another school the next day and they can each have their own sites. They can jump on the net and show their parents the sites that they're doing that day and sort of what they're doing, etc. So that's one way of doing it. It does require an investment. But the interest is there, the enthusiasm is there and because we're providing the training and education and are in touch with the people doing that work, we actually have a handle on the data quality and can monitor and influence that. So that's part of the question, answer I hope. Who's next? There's one more coming first. Hello. I was wondering if I'm correct you were talking about the plugin you created to connect to your national services, ask for capabilities and then show the available layers. For example, WMS layers, am I correct? Yes. If you run GeoNetwork, you can point it at a WMS or WFS server and it will harvest the metadata. That gives us a catalog that can automatically really update itself from the underlying services to provide a current list of what layers are available. The plugin should allow users to search, use a CSW search for matching layers across all the catalogs to store layers as favourites ready for reuse, to cache WFS layers locally if you want to go offline and do some work in the field for instance. And it also, SourcePole added in the capability for WMS layers, you can actually cache those locally, you specify the resolution and coverage you want and it will download and create local tiles so you can go in the field and even use a WMS layer offline. When you come back online again, you can just turn the layers back online and get back to the full functionality. Can this plugin is in the normal QGIS plugin repository? No, it's a work in progress. We have a prototype working. The prototype was actually working on a more of a web scraping than a CSW harvest for the data. We're discussing with Marco from SourcePole here. It's a C++ plugin. Yep. But you are looking at core. Could you repeat that? The plugin is a C++ so it can't go in the Python repository but it can be enabled as part of the core of people that you sort of wanted there. Because I think it's very valuable for a lot of countries but because more and more countries are running CSW services and etc. Yes. Yeah, I couldn't agree more. This might be a leading question. I hope so. It is in two parts. One, you take data from the public. You allow them to contribute. Weren't people at your department afraid that people would put in shit to say it frankly? The next part of the question is how many times have you actually seen people trying to destroy your data set? How do you mean misuse? Deliberately misuse? Never. That's not a big answer. I have no doubt that eventually somebody will, human nature being what it is. But what I have found, this is getting into the whole citizen science issue which is a slight aside but we've got a couple of minutes to go. In one of the New Zealand, is anyone familiar with Nature Watch New Zealand is using iNaturalist? Is anyone familiar with the iNaturalist platform? Okay. It's an open source tool developed in the states for bio observations for citizen science. Users can log on, they can create their own project with their own data fields and you have little bio blitzes run where people go to a section of river and they'll see what they can find and report there and they'll have a look, maybe the same thing next year and you're monitoring trends etc. So it's very citizen science oriented. In New Zealand, just north of Christchurch you'll find a huge number of bat observations. And it's not because the bats are there more than anywhere else, it's because New Zealand bat expert is retired and that's where he lives. So be careful how you use these data because those issues are always there. But anyone going to put the effort into seriously adding significant volumes of data has a genuine interest and commitment to do so and they're unlikely to deliberately try and screw anything up. Thank you. Any other questions? Then I thank you very much Brent for being here and for sharing with us all the way. Thank you. And I'm positive, I'm sure that you win the award for furthest traveler here. There's a few from New Zealand here. There are, yeah, cool. Thank you at lunch.
 People, please have a seat if you want.  There are some seats left here.  So we're moving to the last presentation on this block on Open Data.  It's Brent Wood and he's going, well, as you can see, it is most excellent tourist map.  You should have a look at that before you leave here.  It is really good.  Open Data through Open Standards and Open Source.  There you go.  And you have to use the microphone for the stream.  So otherwise people at home can't hear you.  Yep, they might enjoy that.  I'm from New Zealand, so I've come just about as far as anyone can come to get here.  So yes, greetings from New Zealand and no, that does not have an EPSG code.  Start with apology and then ask a favour.  And I apologise for that, but I wasn't responsible for the schedule.  And my accent, and we tend to talk quickly and people don't understand.  If you don't understand, we're both wasting our time and I've come a long way to do this.  So please wave, do something if you don't understand me so I can slow down and explain.  If you do that, lots of more polite people will be thinking, thank you so much anyway.  So the real story.  Newa, the National Institute for Water and Water Research, pays my wages.  And we study water, atmosphere, climate, fisheries, oceanography, atmospheric science.  We're one of a government agencies which operate as a private business.  So while we are only shareholder as a government minister, we're required to operate and give  that minister a profit to keep his government happy for the financial year.  I'm going to discuss the Open Data Initiative in New Zealand.  The fact that open data is fundamentally about data reuse.  Once you've actually released your data, what happens then?  Some of the things we've done to sort of meet those needs and what I see as a more complete  solution for an open data package.  And most of that's going to be boring text-based stuff and there's some pretty pictures to  illustrate things at the end.  So lots of text.  People who write these things are paid by the syllable.  2011, New Zealand announced a declaration of open transparent government which required  – you don't have to read all that, you get the just of it.  Core government departments and ministries were directed to make all high-value government  data publicly available.  Other agencies that weren't core government were encouraged to make their data available.  Some of the first outputs came out was NZ Goal.  It's a New Zealand government open access licence.  It's a creative, common space framework for agencies to release their data under and consistent  licensing framework tends to make the legal sides of interoperability flow quite nicely.  A very recent enhancement to that was the NZ Goal SE which is a software extension which  recognised that code developed by taxpayer funded projects should be made available just  as data is.  And so we now have an extension to the NZ Goal framework covering code to be released under  an open source licence.  What was particularly inspiring about that was that the initial draft from the lawyer  was put out for public comment.  The result of that was putting it on Lumio.  All the open source enthusiasts jumped on it in a big way and the resulting document  is a genuinely collaborative effort from the public and the open source community into  the government process.  If you're going to release your data, the whole point of releasing it is enabling people  to reuse it.  No one's waving yet.  You can all understand me okay.  One of the problems we went into were there were very few guidelines on how to release  your data.  We didn't do what the Daniels so sensitively did and set up a nice central organisation  that was funded to manage all this.  We told a lot of agencies to do it and didn't tell them how.  And this was supposed to result in interoperability and commonality across all sectors which  was a bit ambitious.  Land Information New Zealand had their Spatial Data Infrastructure Cookbook which recommended  OGCWFS for Spatial Data.  That was about it for government documents describing how agencies should release their  data.  What Niwa and a lot of other agencies in the environmental space did was we created, we  got together, had some discussions and created the Environmental Observation Data Protocol  which covers all our sensor-based stuff, all our bio-data-based stuff where people see  things, observe things, measure things in the environmental space.  And that was across government agencies, the research agencies, regional councils, etc.  So it was a very collaborative exercise and it came up with a very much OGC-based approach  to every agency making their data available.  And the OGC standards are there but we also had things the OGC standards don't cover  such as vocabulary servers, small link data stuff, getting into the bio-data with IPT,  the GBIF standard which I'll touch on later.  So we had some extensions to the core SOS capability.  And agencies already had lots of tools and because we settled on industry standards,  most of those tools could already do, provide the data using the services that we'd mandated.  Lyns introduced last year an institutional GIS for 4,000 people across New Zealand.  That's land information New Zealand I mentioned before.  The tool they chose is an Australian tool from a company that used to be called Digital  Mapping Solutions called Intramaps.  Anyone heard of it?  One, two.  Okay, Intramaps is based on Map Server, QGIS, sits on top of PostGIS.  Some open source stack sold and supported as a commercial application.  They do a fair amount of QGIS development and other work in the space as well.  That's now a nice mix of commercial and FOSS by the major government agency in New Zealand  working with spatial data.  There's a fairly strong vote of confidence in the open source stack there.  So we've got the catalog service, we've got discovery addressed, we've got the other  services for delivery.  We've made our data available.  We can all sit back and go home now and let people do whatever they want.  The use of feedback was, what do we do with it?  Your usual public user doesn't know about GIS.  There was a conference in the US some years ago where the secretary took the notes by  hand and all these references to 516 were in her notes.  That's your GIS.  She'd actually misinterpreted it because you had no idea what GIS was.  And most of the people who are using our data don't know what GIS is.  So my take on open data, if you're going to make, provide it and facilitate reuse, you  need to enable users to do that reuse.  And the software tool that does that is called GIS.  So we have a program now based on QGIS.  We take the free tool, we customise it, we enhance it, we pay the devs to do things with  it.  And people can now access our data, download a tool and actually use that data.  And as part of that process, we're working with DMS again.  We're running training sessions around the entire country for people who are interested.  It's not free, it's commercial operation.  A one-day training course will cost you $200 New Zealand Dollars.  That's pretty cheap for a one-day workshop.  This is leading to a process I'm describing as democratisation of GIS.  And if you've been around as long as me, my first papers were written long-hand on paper.  And they went to a typing pool.  That typing pool was a team of domain-specific experts providing a service to an institution,  which describes your GIS team of yesterday.  What Lyns did, there's now 4,000 people with GIS on their desktop.  It comes when they start with the organisation.  Along with Word, Excel, a PDF reader, you get QGIS.  We have open data.  We have a tool, and we all know there's lots of people out there wanting to use that data,  and we're now enabling to do that.  So some of the tools we've put together, IPT, GBIF.  Anyone familiar with GBIF?  Global Bio Data Information Framework.  If someone in New Zealand wants to find out where all the museum specimens or observations  of a particular species are, you'd think they'd go to a New Zealand dataset, but you wouldn't.  If you were here on a fish called Scaly Gernet, for example, don't ask me why, but there's  a museum at the University of Kansas, a long way from New Zealand, a long way from the  sea, that has a Scaly Gernet from New Zealand in it.  If you want to find all those records, you can't go to an institutional catalogue.  You go to a global catalogue that all those institutions upload the data into, and you  will find all the records that not just the New Zealand institutions hold, but every institution  in the world pretty much.  And IPT is the protocol that GBIF provides for uploading data.  So you have automatic access to any GBIF provider straight from QGIS.  Direct to, it's a river environment classification, 800,000 reaches in New Zealand online as a  WMS.  Our freshwater fish database plugin, Direct Access, we're working on a SOS plugin for  sensor access to data.  And we've set up through the New Zealand Open Source Society a geo network instance in the  last month.  That's being set up to harvest every New Zealand OGC server we can find with a QGIS  plugin that will access that server via CSW and allow users to find and store as favourites  any web service they can track down.  So that's trying to sort of address through the community a sort of consolidated discovery  capability.  So the complete solution is not just data or access to data, but it's tools to allow  you and support you and educate you and train you in how to reuse that data.  And as the last very good presentation identified, most of the users were Joe Block's public.  And I'm looking at the friends of a local stream who are doing an environmental cleanup  of their stream, they're recording water temperatures, they're recording the animals  and plants that are living there.  That's going into databases that other people can then use and share and learn from.  We have sort of botanical societies, we have birdwatches, we have huge amounts of data  going in to little NGO community clubs systems that can now be part of a framework and they  are enabled by adding all this information to help them direct their efforts.  That wasn't supposed to happen.  He just took a while, but I don't know why this did that.  Anyway, GBIF, through our IBT plugin.  This is a pretty picture bit, so a bit more interesting.  Look about five minutes to download the data.  There's a lot of them.  Where you see one point, there's usually lots of them underneath it.  But that allows instant access to data that is not normally available within a GIS framework.  The IBT protocol is not a GIS, not technically a spatial framework.  It uses a Darwin core standard and that data is now available to use within the GIS framework  if you use QGIS.  Within NIWA, one of the first things we're doing about making data available is making  data available to our own staff.  We spread across the country, we have different departments.  Our left hand doesn't know where there is a right hand, let alone what it's doing.  These tools now allow people both in the institution and outside the institution to look at institutional  databases and access and visualise and work with the data.  So we have 40 years worth of fisheries surveys around the New Zealand coast.  You can look at spur dogs, you catch around here as well, they're a global species.  So there's one survey or you can look at different species from multiple surveys.  Data is available, it's all web servers based.  This was not a plugin, we knew our fund was some enhancements, basically allowing you  to do things like, which button gives me my light, that one.  So nice annotated frameworks, a lot of our stuff's maritime, so we want sort of degree  labels because that tends to be on the sea.  So we paid the developers to enhance the print composer so you can do nice maps like that.  One of the fun things with the polar map, which weighs north.  That's using a custom projection because Antarctic Polar Stereographic is actually 180 down.  And if I plot that, it looks wrong.  It's not, but it looks wrong.  So we've got a custom projection with 180 up instead of zero and a nice looking map.  The nice thing about this is our specimen database uses an open source tool called specify  as the collection management tool.  This underlying database is my SQL, non-spatial.  Access is enabled through a GDAL virtual data file.  You're familiar with those?  Essentially it's just a piece of Xml that describes the data source that includes an  SQL to run on that data source.  When you open it in QGIS, it's as if you've opened the underlying data source, not just  the XML file.  The XML file is just a pointer to the underlying data.  So these people who have a collection tool now can use QGIS to interact directly with  that projection tool, generate publication quality graphs directly from it.  And we can open it up via a web service and people can actually, the public can actually  do this and that will be happening.  All of these things are a work in progress.  Our SOS client, genuine collaboration.  We have a Newell plugin in QGIS.  We have a regional council providing a SOS service.  We have a base map provided by yet another CRI as a WMS service.  And we can plot their various observation stations on the map, click on one, find out  what observations are available and display and download those observations.  Another work in progress and source polar working on this with us.  One of the big issues with SOS is that every SOS server has a slightly different implementation  and dialect of SOS.  So working through some of those gets interesting.  I mentioned before the REC2, the river environment classification, 800,000 freshwater reaches  across New Zealand.  You can probably see it in the blue lines here.  It's a transparent WMS layer so it loads pretty much instantly.  It's the back end as opposed to just database.  You can open up the tool, click on a reach.  In this case it was down at the river mouth there.  And click the search upstream button and you'll get a WMS rendition of the upstream network.  Which basically means you can stand anywhere and ask if something's here, where could it  come from or if it's going, where could it go.  This was actually part of another tool that we developed and the interest was such from  user feedback that they wanted this as a standalone capability so that was done.  Fisheries research, acoustic stock assessment using eco sounders to count fish.  Which virtual fish, that's the fish there on that one.  We have a commercial database for our imagery that we paid the developer to provide a web  API for.  And QGIS has an action tool which I'll detail slightly later on.  But with the action tool and the web API, the WFS service for all these transit line  features includes the image ID in that web accessible database for images as one of the  fields.  So you can use the image viewer of your choice in a QGIS action, pass it the URL that includes  the image ID.  You can click on any one of 16,000 transits and see a pretty picture of it.  Native QGIS.  Now the one that's a bit of fun, I think is fun.  It's a good open source story.  Back in 2005, version 0.6 of QGIS, we had developed an acoustic database using PostGIS with a  colleague of mine Gavin McCauley who I introduced to QGIS.  He started using QGIS to look at the acoustic data and wanted the ability to open up an  echogram from the map in our echogram analysis software.  So he created at that stage the action tool in QGIS.  He left Niwa and is currently working for Norway to maintain a Scandinavian connection.  And I emailed him to say what we'd just done with the tool that he'd done in the best part  of 10 years before.  He emailed back a screen cap.  He's sitting on a boat north of Norway, due north of Norway.  He's running QGIS, connecting the WFS service, running the action tool that he created to  open up an echogram for a survey that he actually ran when he was working for Niwa.  So New Zealand data opened up in a boat sitting in the North Atlantic by the original developer.  I don't think it's a nice circular story.  So thank you for your time.  That's sort of what we've done and why we've done it and how we're doing it.  And I'm happy for questions.  So, opportunity for questions.  Who can I hand the mic first?  Although Aine Beck.  Thank you for your talk.  Just a question on your opinion on, you mentioned earlier, the initiative for local communities  for the river cleanup or whatever that was.  How do you see striking that balance between data completeness?  So you've got these pretty extensive field and arrays with all the data for potential  sort of academic uploads compared to someone who might not be able to create all those  fields or submit all that data.  So the potential is there for them not to create complete data and yet at the same time  you don't want to overwhelm them with forcing them to submit everything.  Where do you find that balance?  A lot of it is education.  This is a slightly interesting example because we are focusing at this stage on just the  stream one you mentioned.  NEEWA produces what we call a Schmack kit.  It's a stream health monitoring kit that we provide through councils to schools, community  groups.  It has guidelines on what to do, how to do it, the core indicator species to or how  to identify them.  We're on workshops in that.  So any agency that wants to sort of start in that sort of process, we're actually providing  tools and training and facilities and through the councils we're trying to establish something  that's basically like a toy library.  So the kit can be used by one school one day and another school the next day and they can  each have their own sites.  They can jump on the net and show their parents the sites that they're doing that day and  sort of what they're doing, etc.  So that's one way of doing it.  It does require an investment.  But the interest is there, the enthusiasm is there and because we're providing the training  and education and are in touch with the people doing that work, we actually have a handle  on the data quality and can monitor and influence that.  So that's part of the question, answer I hope.  Who's next?  There's one more coming first.  Hello.  I was wondering if I'm correct you were talking about the plugin you created to connect to  your national services, ask for capabilities and then show the available layers.  For example, WMS layers, am I correct?  Yes.  If you run GeoNetwork, you can point it at a WMS or WFS server and it will harvest the  metadata.  That gives us a catalog that can automatically really update itself from the underlying services  to provide a current list of what layers are available.  The plugin should allow users to search, use a CSW search for matching layers across all  the catalogs to store layers as favourites ready for reuse, to cache WFS layers locally  if you want to go offline and do some work in the field for instance.  And it also, SourcePole added in the capability for WMS layers, you can actually cache those  locally, you specify the resolution and coverage you want and it will download and create local  tiles so you can go in the field and even use a WMS layer offline.  When you come back online again, you can just turn the layers back online and get back to  the full functionality.  Can this plugin is in the normal QGIS plugin repository?  No, it's a work in progress.  We have a prototype working.  The prototype was actually working on a more of a web scraping than a CSW harvest for the  data.  We're discussing with Marco from SourcePole here.  It's a C++ plugin.  Yep.  But you are looking at core.  Could you repeat that?  The plugin is a C++ so it can't go in the Python repository but it can be enabled as part of  the core of people that you sort of wanted there.  Because I think it's very valuable for a lot of countries but because more and more countries  are running CSW services and etc.  Yes.  Yeah, I couldn't agree more.  This might be a leading question.  I hope so.  It is in two parts.  One, you take data from the public.  You allow them to contribute.  Weren't people at your department afraid that people would put in shit to say it frankly?  The next part of the question is how many times have you actually seen people trying  to destroy your data set?  How do you mean misuse?  Deliberately misuse?  Never.  That's not a big answer.  I have no doubt that eventually somebody will, human nature being what it is.  But what I have found, this is getting into the whole citizen science issue which is a  slight aside but we've got a couple of minutes to go.  In one of the New Zealand, is anyone familiar with Nature Watch New Zealand is using iNaturalist?  Is anyone familiar with the iNaturalist platform?  Okay.  It's an open source tool developed in the states for bio observations for citizen science.  Users can log on, they can create their own project with their own data fields and you  have little bio blitzes run where people go to a section of river and they'll see what  they can find and report there and they'll have a look, maybe the same thing next year  and you're monitoring trends etc.  So it's very citizen science oriented.  In New Zealand, just north of Christchurch you'll find a huge number of bat observations.  And it's not because the bats are there more than anywhere else, it's because New Zealand  bat expert is retired and that's where he lives.  So be careful how you use these data because those issues are always there.  But anyone going to put the effort into seriously adding significant volumes of data has a genuine  interest and commitment to do so and they're unlikely to deliberately try and screw anything  up.  Thank you.  Any other questions?  Then I thank you very much Brent for being here and for sharing with us all the way.  Thank you.  And I'm positive, I'm sure that you win the award for furthest traveler here.  There's a few from New Zealand here.  There are, yeah, cool.  Thank you at lunch.
foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_0_215.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_216_21275.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_21276_30947.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_30948_31151.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_31152_31211.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_31212_31619.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_31620_31979.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_31980_32507.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_32508_33467.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_33468_33707.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_33708_33875.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_33876_33923.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_33924_34031.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_34032_34523.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_34524_34991.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_34992_35183.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_35184_39995.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_39996_40067.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_40068_40523.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_40524_40895.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_40896_40943.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_40944_40991.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_40992_41339.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_41340_41747.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_41748_42287.png foss4g-2016-1048-open_source_as_part_of_an_open_data_initiative-hd_42288_42437.png