Evaluation of the semantic research data management system CaosDB in glaciology
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Schlemmer, Alexander
Wörden, Henrik tom
Freitag, Johannes
Fitschen, Timm
Kerch, Johanna
Schlomann, York
Hornung, Daniel
Weikusat, Ilka
Parlitz, Ulrich
Wilhelms, Frank
Lutherm Stefan
We evaluate the open source software CaosDB for its employment in glaciological ice-core research at the Alfred Wegener Institute in Bremerhaven. Here, the system is used for the logistical management of up to several kilometres long ice cores which are stored as ice-core sections in a freezer storage facility. We provided a loan work flow to facilitate and control the movement of ice samples during ice-core processing. Furthermore, we integrate data analysis algorithms for various processing procedures on the ice-core samples. The application of our research data management system (RDMS) to ice-core research demonstrates in particular how a flexible RDMS with a semantic data model can efficiently improve collaborative research.
 Today I will talk about an evaluation of Caustic B in Glaciology. So my talk will be split into two parts. First I will give a quick overview of Caustic B. And in the second part I will talk about how we transferred that system to a problem at the Avi in Bremerhaven. And you might notice there's a third affiliation appearing the Indies scale game we have. We recently founded, some of the authors founded a service company for providing commercial support and custom developments for that system. So what is Caustic B? First of all I would briefly like to introduce some important requirements for us in research and for research data management for our research. So we are interested in having a simple but expressive search functionality. We need to be able to store every data format at any file size. We also need to be able to have the possibility to store link and retrieve raw data but also process data, analysis results and documentation. So not limited to raw data. And we want to support all kinds of data analysis software from simple scripts to high level software. We want a minimally invasive workflow so the last two points are actually related. We want that users don't have to adapt to a new system but we want to adapt the new system to the users and how they work. And one requirement is very important for us. The scientific environments, especially I come from biomedical physics, change really often. We have new devices, we have new hardware, new software. So we have the need for flexible data models. So we cannot reprogram the system every time we buy new hardware or new software. So I think the software landscape has a lot of solutions for data acquisition and for data publication. So for example in the field of electronic lab notebooks there are many good solutions already available. Also in the field of data repositories for the final publication of data there are a lot of really good working possibilities, for example also at the RV. But in an intermediate stage, as well as more targeting the intermediate stage where data analysis kicks in and where we also want to be able to directly store the results of our data analysis again in chaos to be. So not only the raw data but also every intermediate step until we finally push that stuff or the final result to a data repository. And how does it in principle work? So for the data acquisition as I said we want you to use your desired workflow. You probably generate some kind of data files on some kind of file system and KOS TV has an automatic crawler that needs to be adapted to special use cases but in principle is able to automatically index files on the file system in order to create entities and records in KOS TV. And afterwards you are able to either browse the data through a web user interface or use any of the available client interfaces that communicate via RESTful. It's an ad protocol with the server software and here I depicted the Python interface which currently is the most established interface. You can use it for automatically querying data and also for interactive data analysis. So the current state and the future of the project in general is a scientific project as the research group Biomedical Physics at the Max Planck Institute for Dynamics and Self-Organization in Götting is developed since approximately eight years and running stable since 2016. Originally it was developed for our own purposes but we have released it as open source software last year. And as I mentioned we just a few weeks ago founded this service company and KOS TV is currently tested in other workgroups in and outside of Götting. So far we have received very positive feedback and about one of these evaluations I will talk about now. So KOS TV in Glaciology at the Avi in Bremerhaven. So what were the aims of the collaborative project? So initially they had a very concrete problem. They wanted a system for efficient management of I-square boxes and they want to have the possibility to have flexible extension for example to sample management. So to really analyze, to store the analysis results and the samples within the database and connect them. For us it was interesting to test whether this flexible data model really works outside of our own context or in the context of polar research. And we also wanted an evaluation of the system in general in a different scientific environment with possibly different backgrounds from scientists and technicians. And of course very valuable for us feedback from very new users that haven't used the system before. So what are we actually talking about? These ice cores look like this. So this is the state when they are basically drilled in Greenland or Antarctica or in Glacia. And here we can see this process during the actual data acquisition. Of course it's not possible to store them like this in Germany. So as soon as they are transferred to Germany there are huge piles of pallets within a freezer storage facility. Here we can see a pile of boxes. Every one of these boxes contains pieces of ice cores, chunks of ice cores. And of course it's very difficult to keep track of these ice cores especially when they are given away for data analysis to other institutes or also at the same institute for people carrying out the data analysis. So we started designing a very simple data model and integrating it into KSDB. So we essentially have boxes, we have a box type, we have loans which store the properties of boxes that are given away for some amount of time. And we have pallets which are connected to the boxes and basically store the location within freezer facilities. And of course person, a very simple type that can be used to look for all kinds of responsibilities. So this is the initial model. Later I'll show an extension also to sample management. But first of all it allowed us to write away with the existing system, do very simple but useful searches like finding specific records, box with for example a specific number or generate overviews over boxes with specific contents or to find boxes which reference other entities like which reference another pallet with a specific number. So the system in this case already proved that it can handle these rather simple but useful requests right away. Later I will show that also very complex statements that like these chained queries for linked properties become possible and enable users to search for for example data that is a cost that are connected to specific material or data analysis. So this is how it would look like if we search for a box in the web UI of KSTB. You see essentially three text properties here and three linked properties, a responsible person, a box type and a pallet. This is basically what the visualized representation of what we just saw looks like. And now the important workflow comes into play when we want to borrow a box. So assume we want to borrow this box, do an analysis on this box, we can click this little button here on the top right and fill a form. So in this case I already filled it with my name and my email address and expected return date and some other comments that are useful for the data curator at this institute. And yeah, the system will handle this, will connect another type which we have seen previously alone which stores all the information about the whole process of lending and returning ice core boxes in this process and it will also adapt to what is happening here. So in this case the administrator of the system can accept the borrow request, can also put the box into place and handle the whole process of lending and returning boxes and the system will keep track of the changes. So this is what the whole ice core box loan workflow looks like. So we have basically a borrow workflow with an end state lend and we have another workflow for the return. And there another challenge came into play. So the ice core facility where they store boxes is handled by an external company and these companies as soon as boxes are returned, hands out an excel sheet basically with new information. So with new information about where boxes have been put in the end and on which palette they are stored and it was a bit clumsy to enter all this information about the new location manually. So we decided to integrate an automatic return which we integrated into KSTB. So we have just a small JavaScript extension for KSTB which allows you to upload an excel sheet. So this is the excel sheet provided by the company and if you press submit, this sheet will be uploaded, processed by KSTB and all the information that is necessary to keep track of the new location of boxes will be updated and stored in the loan property. So in the end you get a result that 16 boxes have been updated successfully and this really helps in managing the daily work with the loan workflow at this institute. So for searching the KSTB, of course we have this full text semantic search facility in the KSTB query language but we provide a lot of shortcuts for the end user that facilitates basically all these shortcuts just generate some statements and here is the query language in order to search for statements that are used very often efficiently. And these shortcuts can also update it and create it by the user depending on what is necessary for them. And we found that actually the query language already proved to be quite simple to use. People get adapted to this language really quickly. Here we see also one product that worked right away because it was already integrated that you can for example generate over views over existing entities for example over all current loans with a very simple query. Okay for the final part I want to show that it's actually possible to go to sample management and also to do automatic data analysis from within the system itself and therefore we created a prototype so behind each core there is also a structure that builds on it and these cores are actually cut into strips and separated into backs and into sections and in the end there are different types of analysis run on these sections. And here for example for this fabric type we have two types of analysis that can be carried out and one analysis might result for example in one plot that is later maybe published or used for further data analysis. And we thought it might be useful to integrate this because it's needed very often into the system itself. So for a specific type of analysis where we already have a pre-processed image we provide a button in the user interface called Analyze Data which you can use to feed this data set directly into a Python script that is running in the background on the server and which automatically generates an output and I think what's also really valuable also connects this result directly to the analysis and stores this back into KOSDB. So when user uses this automatic analysis facility they can also directly keep track of processed results of output of results because it's stored into KOSDB back right away. Yeah and with this I would like to thank you for your attention and highlight that we have this open source project on GitLab, on our local GitLab by the Gevvdg in Göttingen and also a paper on Archive which is also currently under review in a peer review journal but which you can already download here as a pre-print. Thank you.
 Today I will talk about an evaluation of Caustic B in Glaciology.  So my talk will be split into two parts.  First I will give a quick overview of Caustic B.  And in the second part I will talk about how we transferred that system to a problem at  the Avi in Bremerhaven.  And you might notice there's a third affiliation appearing the Indies scale game we have.  We recently founded, some of the authors founded a service company for providing commercial  support and custom developments for that system.  So what is Caustic B?  First of all I would briefly like to introduce some important requirements for us in research  and for research data management for our research.  So we are interested in having a simple but expressive search functionality.  We need to be able to store every data format at any file size.  We also need to be able to have the possibility to store link and retrieve raw data but also  process data, analysis results and documentation.  So not limited to raw data.  And we want to support all kinds of data analysis software from simple scripts to high level  software.  We want a minimally invasive workflow so the last two points are actually related.  We want that users don't have to adapt to a new system but we want to adapt the new  system to the users and how they work.  And one requirement is very important for us.  The scientific environments, especially I come from biomedical physics, change really  often.  We have new devices, we have new hardware, new software.  So we have the need for flexible data models.  So we cannot reprogram the system every time we buy new hardware or new software.  So I think the software landscape has a lot of solutions for data acquisition and for  data publication.  So for example in the field of electronic lab notebooks there are many good solutions  already available.  Also in the field of data repositories for the final publication of data there are a lot  of really good working possibilities, for example also at the RV.  But in an intermediate stage, as well as more targeting the intermediate stage where data  analysis kicks in and where we also want to be able to directly store the results of our  data analysis again in chaos to be.  So not only the raw data but also every intermediate step until we finally push that stuff or the  final result to a data repository.  And how does it in principle work?  So for the data acquisition as I said we want you to use your desired workflow.  You probably generate some kind of data files on some kind of file system and KOS TV has  an automatic crawler that needs to be adapted to special use cases but in principle is able  to automatically index files on the file system in order to create entities and records in  KOS TV.  And afterwards you are able to either browse the data through a web user interface or use  any of the available client interfaces that communicate via RESTful.  It's an ad protocol with the server software and here I depicted the Python interface which  currently is the most established interface.  You can use it for automatically querying data and also for interactive data analysis.  So the current state and the future of the project in general is a scientific project  as the research group Biomedical Physics at the Max Planck Institute for Dynamics and  Self-Organization in Götting is developed since approximately eight years and running  stable since 2016.  Originally it was developed for our own purposes but we have released it as open source software  last year.  And as I mentioned we just a few weeks ago founded this service company and KOS TV is  currently tested in other workgroups in and outside of Götting.  So far we have received very positive feedback and about one of these evaluations I will  talk about now.  So KOS TV in Glaciology at the Avi in Bremerhaven.  So what were the aims of the collaborative project?  So initially they had a very concrete problem.  They wanted a system for efficient management of I-square boxes and they want to have the  possibility to have flexible extension for example to sample management.  So to really analyze, to store the analysis results and the samples within the database  and connect them.  For us it was interesting to test whether this flexible data model really works outside  of our own context or in the context of polar research.  And we also wanted an evaluation of the system in general in a different scientific environment  with possibly different backgrounds from scientists and technicians.  And of course very valuable for us feedback from very new users that haven't used the  system before.  So what are we actually talking about?  These ice cores look like this.  So this is the state when they are basically drilled in Greenland or Antarctica or in Glacia.  And here we can see this process during the actual data acquisition.  Of course it's not possible to store them like this in Germany.  So as soon as they are transferred to Germany there are huge piles of pallets within a freezer  storage facility.  Here we can see a pile of boxes.  Every one of these boxes contains pieces of ice cores, chunks of ice cores.  And of course it's very difficult to keep track of these ice cores especially when they  are given away for data analysis to other institutes or also at the same institute for  people carrying out the data analysis.  So we started designing a very simple data model and integrating it into KSDB.  So we essentially have boxes, we have a box type, we have loans which store the properties  of boxes that are given away for some amount of time.  And we have pallets which are connected to the boxes and basically store the location  within freezer facilities.  And of course person, a very simple type that can be used to look for all kinds of responsibilities.  So this is the initial model.  Later I'll show an extension also to sample management.  But first of all it allowed us to write away with the existing system, do very simple but  useful searches like finding specific records, box with for example a specific number or  generate overviews over boxes with specific contents or to find boxes which reference  other entities like which reference another pallet with a specific number.  So the system in this case already proved that it can handle these rather simple but  useful requests right away.  Later I will show that also very complex statements that like these chained queries for linked  properties become possible and enable users to search for for example data that is a cost  that are connected to specific material or data analysis.  So this is how it would look like if we search for a box in the web UI of KSTB.  You see essentially three text properties here and three linked properties, a responsible  person, a box type and a pallet.  This is basically what the visualized representation of what we just saw looks like.  And now the important workflow comes into play when we want to borrow a box.  So assume we want to borrow this box, do an analysis on this box, we can click this little  button here on the top right and fill a form.  So in this case I already filled it with my name and my email address and expected return  date and some other comments that are useful for the data curator at this institute.  And yeah, the system will handle this, will connect another type which we have seen previously  alone which stores all the information about the whole process of lending and returning  ice core boxes in this process and it will also adapt to what is happening here.  So in this case the administrator of the system can accept the borrow request, can also put  the box into place and handle the whole process of lending and returning boxes and the system  will keep track of the changes.  So this is what the whole ice core box loan workflow looks like.  So we have basically a borrow workflow with an end state lend and we have another workflow  for the return.  And there another challenge came into play.  So the ice core facility where they store boxes is handled by an external company and  these companies as soon as boxes are returned, hands out an excel sheet basically with new  information.  So with new information about where boxes have been put in the end and on which palette  they are stored and it was a bit clumsy to enter all this information about the new location  manually.  So we decided to integrate an automatic return which we integrated into KSTB.  So we have just a small JavaScript extension for KSTB which allows you to upload an excel  sheet.  So this is the excel sheet provided by the company and if you press submit, this sheet  will be uploaded, processed by KSTB and all the information that is necessary to keep track  of the new location of boxes will be updated and stored in the loan property.  So in the end you get a result that 16 boxes have been updated successfully and this really  helps in managing the daily work with the loan workflow at this institute.  So for searching the KSTB, of course we have this full text semantic search facility in  the KSTB query language but we provide a lot of shortcuts for the end user that facilitates  basically all these shortcuts just generate some statements and here is the query language  in order to search for statements that are used very often efficiently.  And these shortcuts can also update it and create it by the user depending on what is  necessary for them.  And we found that actually the query language already proved to be quite simple to use.  People get adapted to this language really quickly.  Here we see also one product that worked right away because it was already integrated that  you can for example generate over views over existing entities for example over all current  loans with a very simple query.  Okay for the final part I want to show that it's actually possible to go to sample management  and also to do automatic data analysis from within the system itself and therefore we  created a prototype so behind each core there is also a structure that builds on it and  these cores are actually cut into strips and separated into backs and into sections and  in the end there are different types of analysis run on these sections.  And here for example for this fabric type we have two types of analysis that can be  carried out and one analysis might result for example in one plot that is later maybe  published or used for further data analysis.  And we thought it might be useful to integrate this because it's needed very often into the  system itself.  So for a specific type of analysis where we already have a pre-processed image we provide  a button in the user interface called Analyze Data which you can use to feed this data set  directly into a Python script that is running in the background on the server and which  automatically generates an output and I think what's also really valuable also connects  this result directly to the analysis and stores this back into KOSDB.  So when user uses this automatic analysis facility they can also directly keep track  of processed results of output of results because it's stored into KOSDB back right  away.  Yeah and with this I would like to thank you for your attention and highlight that we  have this open source project on GitLab, on our local GitLab by the Gevvdg in Göttingen  and also a paper on Archive which is also currently under review in a peer review journal  but which you can already download here as a pre-print.  Thank you.
Alexander_Schlemmer_0_587.png Alexander_Schlemmer_588_899.png Alexander_Schlemmer_900_2867.png Alexander_Schlemmer_2868_4175.png Alexander_Schlemmer_4176_6455.png Alexander_Schlemmer_6456_6551.png Alexander_Schlemmer_6552_6599.png Alexander_Schlemmer_6600_7979.png Alexander_Schlemmer_7980_8555.png Alexander_Schlemmer_8556_9107.png Alexander_Schlemmer_9108_9431.png Alexander_Schlemmer_9432_12563.png Alexander_Schlemmer_12564_13115.png Alexander_Schlemmer_13116_13955.png Alexander_Schlemmer_13956_17987.png Alexander_Schlemmer_17988_18143.png Alexander_Schlemmer_18144_18851.png Alexander_Schlemmer_18852_18983.png Alexander_Schlemmer_18984_20562.png