Towards a generalised approach for defining, organising and storing metadata from all experiments at the ESRF.
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Götz, Andrew
After more than 20 years of operation the situation concerning metadata at the ESRF is still very disparate between beamlines. The way the metadata required to analyse the raw data are defined and collected depends largely on the beamline concerned. Approaches vary from fully automated solutions implemented on the MX beamlines to a combination of automated and manual collection of metadata. This talk will not present the solution for MX (see talk by Gordon Leonard for more info) but will present a new approach for automating the collection and storing of well defined metadata for all experiments. The solution is based on a generic tool built at the ESRF which uses HDF5 for file format, Nexus for definitions (where possible) and ICAT for the metadata catalogue. The talk will present concrete examples of its use for nano tomography and fluorescence and radiation therapy. Ongoing work on how this will be extended to small angle scattering, coherent diffraction and eventually all other techniques will be presented. The talk will conclude with a discussion on the role of metadata in data policy and management.
 Okay, good afternoon. My name is Andy Goetz from the ESRF and I'm not a crystallographer, so we're going to leave the crystallographic world and the MX world and I'm going to talk about the other fields and also represent a source, I think. Tell you what we're doing at the source, yeah, or planning to do, because this talk is about very recent work and so we're moving towards something which we hope will solve a number of the issues which have been raised here. So before I start, well, I thought I'd wanted to just show a few pictures, pictures by an artist who I discovered when I was preparing this talk called Cynthia Gregg, who works on various things, but this collection called Life Size reminded me of what we were doing here in a way that I'm going to give you a very detailed account of what we're trying to do in terms of software, but at any point in time you can take another step back and just look at the pictures or try and get the overall viewpoint of what we're working on. When I did contact her, if I could get her permission to use these pictures and she said no problem at all and checked out what we were doing and said it sounded very fascinating. I don't know if she would say the same if she was here now. So just to remind you ESRF, European Synchrotron, where are we today? In terms of metadata, my rapid summing up is in MX, we're in a very good situation, but then for the rest, it's very, very, I mean, some metadata is still collected manually, some of it's not collected, some of it's very little of it's archived and so on. And that's what we're now trying to address. And we are in the throes of doing a major upgrade of the source as well, to get even more brilliance. And I don't know if you'll like that, but anyway. Just to show you what we're dealing with in terms of raw data, this is the last 12 months until June before the summer shutdown. You can see the shutdowns are where you have no data. And we're dealing with at the moment one point about 1.2 petabytes of raw data produced a year. And this is going up as usual. What was interesting for me from this graph is it's actually linear, it's not exponential. It used to be exponential, but now it's more linear. And why is this? Because of all these detectors, a number of them come from the sponsor, of course, but these are not our worst, let's say, our most challenging detectors, the most challenging ones comes from other suppliers. And we have to deal with all of these. That's why we have a lot of data and metadata. And we also have to deal with new detectors every year. And I think one issue that hasn't been touched upon yet, or I don't know if someone will, I won't have the time is the whole issue of calibration. Calibration of these new detectors who have multiple panels on 3D, have a lot more, bring on a lot of challenges every year. And we keep on being faced with this. And that's also part of the metadata. So if you look around what people mean when they say metadata, it's very hard to come up with a very, with a common definition. And there are a few interesting quotes I found on the web is often we just talk past each other because we don't mean the same thing. Often this generic term data about data is used. I tried that on my wife last weekend because she wanted to know what the talk was about. She just got this blank stare when I said data about data. It's not an obvious definition. So I've defined it, or we've defined it in the context of this project, is all data which is needed in addition to the raw data to analyze the data. So it's all about being driven, high priorities being given to the analysis programs. And I think just aside, one of the reasons why metadata is badly defined is because if you look at this dictionary entry, you could say every entry, the name of every entry is metadata. But on the other hand, you could also take all the entries and just use that as data. So I think metadata has this strange kind of dualities. It can be data for one person and metadata for another person. Whereas raw data is very rarely metadata. So I think that's why there's this confusion, why we don't agree always. So I've identified three, we've identified three classes of metadata. And the one that's been picked up by the two fingers there is the one that we now are trying to solve. The three classes that we're talking about are beam line and sample, everything that goes in to start your experiment describing that. The experiment itself, everything that happens during the experiment. And then the results which are usually the models that you're going to develop and then to interpret what has been measured. The one that I pointed out there with the arrow on it, this is the one that we will be dealing with here in this talk. So and if you think of three classes of metadata, we basically talk about three different kinds of databases. There's the database which is on the beam line, the one that's on the left. This is often the beam line archive, could be your EPIX archive or your Tango archive. And that's the one that's usually got the name of the VME in there which has no significance for the data analysis normally. That one has been heavily, let's say, assisted by the Nexus class definitions. I think a lot of it came from how are we going to describe the entire instrument from an archiving point of view. What we're not dealing with is the metadata for analyzing the data. And these are in databases like ICAT, this is the one that we're using, or ISPyB which is another one that's been used on the MX beam lines. ISPyB is a bit between both of them. And Nexus has defined some things there but many things are not defined because it's dependent on analysis programs. And that's where we have to get a handle on things. And the third catalog which I won't deal with is the one that you know probably much better than I do is the PDB type, the ICSD, all these different catalogs where you put your results in. But those are usually external to the sources. I would say, I mean my own reading of what we've been doing since 20 years is we don't manage metadata very well. And when I look at the other sources I get a similar feeling as well. And when I hear you people I get a similar feeling. I think the reason for that is that we've been trying to, we've left the job in the hands of the beam lines in the sense of most of the time the people who in my group who do the data acquisition know nothing about the analysis and are never going to analyze the data. The people who analyze it usually say we depend on what's recorded. And the beam line scientists is so busy or has, I mean they have so much to do you can't expect them. I mean I think the 20 years as just shown as they not sufficiently resourced and sufficiently, let's say it's not the main job, is to define these kind of metadata standards. And to make sure that this is complete and from a computing aspect this is well managed. So I think it's been, I mean we've given the job or relied on the wrong people. And what we need to do is not to join forces. It has happened where you have a number of beam lines, we have the critical mass like in MX, ESRF, there's enough interest there and resources that you can address the problem in a complete way. But for all the other techniques which is the one that I'm trying to address in this talk, I mean it's often one person or one person and a few postdocs and that and the communities are very, the user communities are very heterogeneous here. So the new approach is to say that we're going to do this from a global point of view. We're going to do this for all beam lines. This means that the computer guys like myself and people in my team are going to come, are going to build a solution and this solution will be then put on the beam lines. This will define a number of things which are common and then ask for the scientists to provide the information that's lacking. So in a way it's a centralized approach but getting the information locally. And the results of that are as follows up to now. First of all I can, we've got a list of requirements if you like. The requirements are to collect metadata for all beam lines instead of saying let each beam line collect their own metadata. We have to store this metadata forever because there's no reason not to do that and make it available via a permanent identifier like DOI for people to use. And this will allow us to offer more services which I'll come to later and even provide the management with the possibility of defining the data policy for all beam lines as well. We have a second remit if you like. It's moving to advanced metadata management and this is the case of MX. And there's certain experiments who can also go into that area where the technique is so well defined that often you can imagine doing this, doing much more than you're doing for everyone. And for this there's a talk tomorrow which is by Gordon, Gordon Leonard who will be talking on this at this workshop and the word there is often called I spy B. I spy B is just the technology or the implementation. Gordon will talk about the metadata needs as well. So I want to talk about MX. What we've done has been built up all entirely on existing standards so we didn't want to reinvent anything. We haven't reinvented Nexus nor HDF5. We took an existing metadata catalog from SDFC. We have I spy B and of course where things overlap with the SIF format we will use that as well. This slide summarizes everything that we've done and how it works. So if you look over here it's not very let's say the writing is a bit small so you'll have to read the slides afterwards. But basically we constrain on a site-wide basis the definition of metadata. Then we link these definitions of course we add definitions when they're missing. We link these definitions to what's on the beam line or in the environment. Then during the experiment this metadata is collected and finally it's ingested into an HDF5 file which is given to the user, the master file I'll come back to that. Eventually it'll go into an e-log book. This is not done yet and then goes into the metadata catalog as well. So it's all this approach can be applied to all beam lines. You can have also the algorithms as well, hard works, the flowchart is basically we install some software on the beam line. We ask the scientists what metadata do they need. We show them what's available already, what we've already got predefined. We add the missing information, put it back in the global definition so it's available for everyone and then we configure this on the beam line. This is done with a number of tools with graphical user interfaces. Over here you see everything that's defined in iCat. All of the nexus classes over here and you drag and drop the nexus classes into the global definition and then link that to where it's stored in the database. This using a tool like this and a global configuration file is everyone starts off from the same point and secondly you're constraining the user not to invent things that don't exist already. If you just zoom on this you should be able to read all of these in the slides and you can see how this has been used on the real example at the ESRF. Today we have a number of global definitions and we're extending this for specific techniques. This is microbeam radiotherapy, tomography, scanning, measurement, generic scanning. This is really only the beginning and this will be extended. The idea is to extend this to all the beam lines. Once you get to the beam line you configure it, you configure the beam line based on what is defined in the configuration file of what metadata are already defined for the site and then during the experiment this is recorded and sent to an HDFI file as well as to iCat and at any time afterwards you can regenerate this file if someone needs it. One thing that's key to this whole thing and that's where we differ with when I see people talking a lot about image formats or that, the headers in image files are not sufficient at all. There's a lot of the context of what makes a data set and if you look at other techniques they have much more varied ways of collecting data. They do multiple scans, sub-scans, in the scan they do calibration runs and things like that. All of this contextual information cannot be put in a single image header file so you need a global file to do that and there the scan is really the key concept that defines what is a data set. It also has to be very flexible as well. You have to be able to mix multiple techniques in the same file because people are inventing new techniques, the techniques which exist only on one beam line and I don't think we can expect international standards on those. We're going to have to be flexible in accepting and defining groups of data which don't exist elsewhere. This is for instance the input we got from the nano beam line where we're working on holography tomography where they take multiple tomography data sets at four distances and then they combine that together. The guy says the scientist wants this information that's on yeah once this as metadata. I think this is very specific to the way they do a holography and you can't expect this to be now part of some standard or anything but you have to put this in the file and you have to make this also available. So this is the kind of data that we're dealing with and where I think this kind of flexibility will be required on many beam lines and on all sites as well. If you look at tomography what we've got here is these things again are all the information that's required for the data analysis and very specific to these data analysis programs. So nothing that can be that would be hard to make an international standard out of that. If you look at what they're doing in radiotherapy they don't have data analysis programs they don't analyze the data they have patients and they have to keep a protocol of certain parameters. These have to also be stored as metadata. This can all be captured with the system. It's a generic scan which allows us to treat any kind of multi-axis scan. We can measure include dynamic data as well because metadata was initially defined as static data during the experiment but there's a lot of dynamic data which needs to be captured as well. This is captured in the measurement. So once we get to the beam line there's a lot of shovel work to do to make this hook up everything that we've got. Here you can see the setup for on the right hand side no left hand side. This is what's being defined on the beam line. So it's the scientist said from this technique I need this and this metadata out of the global configuration and on the left hand side you get the this is the description of the beam line hardware. And by dragging and dropping elements across you link up the definition with where the source the data is and then during runtime it just gets a trigger and says start of scan take metadata and this goes into the file and you get a list of entries in an hdfi file and if you drill down in those entries you find you'll find all of your metadata. Our current status is we have this running on two beam lines and work is started started on two more. We're addressing hollow tomography, nano fluorescence, nano diffraction and these techniques and the next beam lines want to do spectroscopy, full field diffraction. I mean the list of techniques I can see is going to be in the hundreds by the end of this something which is going much beyond what just talking about a single technique. We have estimated based on this how much metadata we think we're going to generate at the ESRF. At the moment our estimate in terms of size of the database based is a rough estimate because we're going from a single experiment oh let's say six months of data collected on one beam line all the time. We come up with 120 gigabytes per year for the all the experiments at the ESRF which is a very modest figure and there's no financial reason why we can't store that forever and this has triggered the discussion at the ESRF of the metadata policy. This has always been a very painful still is very painful and hasn't come we haven't come to a conclusion yet but now that we got a handle on all the metadata at the ESRF there's no reason not to actually now come up with this metadata policy which will say which metadata are available for everyone and so we're going to have to take this injection we're going to have to work on this and come up with a site wide policy which has been done at ISIS has been done at ILL but to my knowledge only Elettra I don't know if other synchrotrons have one but we're going to there's a pledge by the directors to do this during the next during phase two that there will be a data policy for all data. The metadata is definitely the key to any kind of improvements we want to offer any kind of services we want to offer because none of these things can be automated if we don't have the metadata and and I think the challenge is collecting the metadata but also doing the calibration as well this has to be done before the experiment or just I mean very close to when you want to analyze the data so all of these kind of extensions to data services will depend on having metadata and we in a way we block today because we can't offer we can only offer it on an individual basis but not on a site wide basis one of the services we were proposing to offer was called data analysis as a service and we even prepared an EU proposal for this for H2020 but we were not successful but just to show you the key role that metadata plays in this is you have all of this cloud structure here which allows you to access and do offer services but everything goes via the metadata catalog. The metadata catalog is what will identify who has access to what data and what what does the data mean so without that we can't offer the service as well and finally how can this work drop help us I think any kind of metadata schemes which are proposed here we'll make sure that we can somehow we'll put them in our system to see that we can use them or see where we can find common ground maybe a metadata policy kind of template or proposal from the IUCR would help sometimes we end up having discussions of should the title of the proposal be public I mean things like that if international organizations say these things are standard then maybe it'll help sites to come up with to get over these these blocking points sometimes I think it would be great if a board like this a working group like this said which must curate raw data and not say you can do it if you want and so on be a bit more prescriptive on this and of course anything that's related to new techniques that are in crystallography because I have the impression yo is very focused on the current techniques in MX but if I think I see what people are doing with zero crystallography with meshing and things like that are these techniques also being covered by the the current definitions I think this I mean there's going to be a lot of evolution I think especially with the new sources we're not the only ones doing this I think Daisy I know Daisy's doing something similar NSLS2 is also doing that you heard from Herb similar thing I really think the key to managing the metadata is having a master file where all the metadata is put together with links to the raw data then so it's not only dependent on what's coming out of the detector but everything around it we hope we're going to know in 20 years time I'll be able to well I'll probably be retired but then someone else will be able to say we've solved this problem and I just think we'll probably be a challenge getting all the scientists a majority of scientists to accept what we propose them in terms of definitions because there's always people I mean things are when you start going into the details it is much more complicated than just what a computer scientist thinks the problem is just the definition of energy in some experiments it can be many definitions okay and the credits are most of this work is the work of my colleagues I'm just presenting it and I'm very grateful also to have access to these photographs which I find were quite pretty okay that was it was faster than that was so with your idea of basically for every experiment having all the metadata in one HDF file is then to take this one step further the idea also to include all the data in HDF files or will they still be stored in whatever way they come out of the detectors the goal is to store the file the data the raw data in HDF files that we can use the linking mechanism from the master file but as you know from the SRF we have some very old data formats we have a number of different data formats those files can still be listed in the master file but you won't be able to use HDF file linking with it the concept that we've taken is we putting the system in place in parallel with the existing system so it means that if people and data analysis programs can adapt when they ready and you'll have both systems running in parallel so you'll have the old files and the old way of storing metadata or not storing metadata plus in parallel the new way thank you Camille Dubek, Lance Florence just a brief comment about this electronic logbook you were talking about and the metadata which are actually not in the system which need to be introduced by the users or the operator they're better and worse organized people and maybe it's worth of making such a kind of table format for that with a checklist just to be sure that in the time of experiments all the needed metadata are stored because otherwise when the experiment is finished they may be lost somewhere lost in the papers and I agree I'm sometimes a bit I find it's exasperating how much how much sloppiness is acceptable you know in a way it seems strange if you with other systems let's say more with older sciences or that things are often much more organized you wouldn't imagine not starting experiment with having put in all of that information but at least the synchrotron I think it's still a young science and so sometimes I'm amazed at how people work here I mean it works in the end but with a lot of human effort I sense that the ESRF senior management is taking a close interest they will have asked you the question what will success look like what was your answer? Success would be being able to provide new services that we don't have today around data and that for all beam lines because at the moment it depends very much on which technique you're talking about and the people involved so if you want to offer a cloud-based service or that or you want to offer archiving or you want to offer automatic processing these services can only be offered if you have the correct metadata so if we if we doing those things which we're not doing today for all techniques it means that we've actually we have succeeded I imagined you might have answered avoiding failure I'd be very I'm it's not so easy to define failure because if I define failure as a situation today it would be a way of saying the synchrotron is not working and it is working and science is coming out of it you can work like that as well but the neutron sources would say every neutron ends up supporting a publication Vladik gave his documentation that 99.7% of synchrotron X-rays for MX do not support a publication so between these two extremes we have here a consistent at least metric for what success would look like which is that with the better metadata capture and we also insisted on raw data but it was softened we would have a much more efficient usage of the photons or neutrons that are produced I would add one comment which was the most the biggest sort of sign of success was when the scientists the beam line on which this has been deployed said it's now become a pleasure to do experiments this is that meant that we had added value to what he was doing very good Simon Cole Southampton so you essentially approach people a pick list of metadata items you've got the whole catalogue and they can generate a shopping list for their particular technique once you've done that many times do you think you'd be able to pick out what are the core terms would you be able to come up with a core schema that would be very useful for people outside of the central facilities world you live in a very constrained environment in central facilities when it goes out into institutions things get messy really messy you think you've got a messy problem this is what happens in universities and so if we had something that we could go on that would be very useful I think there's nothing I mean this is all open we can share this information
 Okay, good afternoon. My name is Andy Goetz from the ESRF and I'm not a crystallographer,  so we're going to leave the crystallographic world and the MX world and I'm going to talk  about the other fields and also represent a source, I think. Tell you what we're doing  at the source, yeah, or planning to do, because this talk is about very recent work and so  we're moving towards something which we hope will solve a number of the issues which have  been raised here. So before I start, well, I thought I'd wanted to just show a few pictures,  pictures by an artist who I discovered when I was preparing this talk called Cynthia Gregg,  who works on various things, but this collection called Life Size reminded me of what we were  doing here in a way that I'm going to give you a very detailed account of what we're trying to do  in terms of software, but at any point in time you can take another step back and just look at  the pictures or try and get the overall viewpoint of what we're working on. When I did contact her,  if I could get her permission to use these pictures and she said no problem at all and  checked out what we were doing and said it sounded very fascinating. I don't know if she would say  the same if she was here now. So just to remind you ESRF, European Synchrotron, where are we today?  In terms of metadata, my rapid summing up is in MX, we're in a very good situation, but then for  the rest, it's very, very, I mean, some metadata is still collected manually, some of it's not  collected, some of it's very little of it's archived and so on. And that's what we're now  trying to address. And we are in the throes of doing a major upgrade of the source as well,  to get even more brilliance. And I don't know if you'll like that, but anyway. Just to show you  what we're dealing with in terms of raw data, this is the last 12 months until June before the summer  shutdown. You can see the shutdowns are where you have no data. And we're dealing with at the moment  one point about 1.2 petabytes of raw data produced a year. And this is going up as usual. What was  interesting for me from this graph is it's actually linear, it's not exponential. It used to be  exponential, but now it's more linear. And why is this? Because of all these detectors, a number of  them come from the sponsor, of course, but these are not our worst, let's say, our most challenging  detectors, the most challenging ones comes from other suppliers. And we have to deal with all of  these. That's why we have a lot of data and metadata. And we also have to deal with new detectors  every year. And I think one issue that hasn't been touched upon yet, or I don't know if someone  will, I won't have the time is the whole issue of calibration. Calibration of these new detectors  who have multiple panels on 3D, have a lot more, bring on a lot of challenges every year. And we  keep on being faced with this. And that's also part of the metadata. So if you look around what  people mean when they say metadata, it's very hard to come up with a very, with a common definition.  And there are a few interesting quotes I found on the web is often we just talk past each other  because we don't mean the same thing. Often this generic term data about data is used. I tried that  on my wife last weekend because she wanted to know what the talk was about. She just got this  blank stare when I said data about data. It's not an obvious definition. So I've defined it,  or we've defined it in the context of this project, is all data which is needed in addition to the  raw data to analyze the data. So it's all about being driven, high priorities being given to the  analysis programs. And I think just aside, one of the reasons why metadata is badly defined is  because if you look at this dictionary entry, you could say every entry, the name of every entry  is metadata. But on the other hand, you could also take all the entries and just use that as data.  So I think metadata has this strange kind of dualities. It can be data for one person and  metadata for another person. Whereas raw data is very rarely metadata. So I think that's why  there's this confusion, why we don't agree always. So I've identified three, we've identified three  classes of metadata. And the one that's been picked up by the two fingers there is the one that we  now are trying to solve. The three classes that we're talking about are beam line and sample,  everything that goes in to start your experiment describing that. The experiment itself, everything  that happens during the experiment. And then the results which are usually the models that you're  going to develop and then to interpret what has been measured. The one that I pointed out there  with the arrow on it, this is the one that we will be dealing with here in this talk. So and if  you think of three classes of metadata, we basically talk about three different kinds of  databases. There's the database which is on the beam line, the one that's on the left. This is  often the beam line archive, could be your EPIX archive or your Tango archive. And that's the  one that's usually got the name of the VME in there which has no significance for the data  analysis normally. That one has been heavily, let's say, assisted by the Nexus class definitions. I  think a lot of it came from how are we going to describe the entire instrument from an archiving  point of view. What we're not dealing with is the metadata for analyzing the data. And these are  in databases like ICAT, this is the one that we're using, or ISPyB which is another one that's been  used on the MX beam lines. ISPyB is a bit between both of them. And Nexus has defined some things  there but many things are not defined because it's dependent on analysis programs. And that's where  we have to get a handle on things. And the third catalog which I won't deal with is the one that  you know probably much better than I do is the PDB type, the ICSD, all these different catalogs  where you put your results in. But those are usually external to the sources. I would say, I  mean my own reading of what we've been doing since 20 years is we don't manage metadata very well. And  when I look at the other sources I get a similar feeling as well. And when I hear you people I get  a similar feeling. I think the reason for that is that we've been trying to, we've left the job in  the hands of the beam lines in the sense of most of the time the people who in my group who do the  data acquisition know nothing about the analysis and are never going to analyze the data. The people  who analyze it usually say we depend on what's recorded. And the beam line scientists is so busy  or has, I mean they have so much to do you can't expect them. I mean I think the 20 years as just  shown as they not sufficiently resourced and sufficiently, let's say it's not the main job,  is to define these kind of metadata standards. And to make sure that this is complete and from a  computing aspect this is well managed. So I think it's been, I mean we've given the job or relied  on the wrong people. And what we need to do is not to join forces. It has happened where you have  a number of beam lines, we have the critical mass like in MX, ESRF, there's enough interest there  and resources that you can address the problem in a complete way. But for all the other techniques  which is the one that I'm trying to address in this talk, I mean it's often one person or one  person and a few postdocs and that and the communities are very, the user communities are  very heterogeneous here. So the new approach is to say that we're going to do this from a global  point of view. We're going to do this for all beam lines. This means that the computer guys like  myself and people in my team are going to come, are going to build a solution and this solution  will be then put on the beam lines. This will define a number of things which are common and then  ask for the scientists to provide the information that's lacking. So in a way it's a centralized  approach but getting the information locally. And the results of that are as follows up to now.  First of all I can, we've got a list of requirements if you like. The requirements are to collect  metadata for all beam lines instead of saying let each beam line collect their own metadata. We  have to store this metadata forever because there's no reason not to do that and make it available  via a permanent identifier like DOI for people to use. And this will allow us to offer more  services which I'll come to later and even provide the management with the possibility of  defining the data policy for all beam lines as well. We have a second remit if you like. It's  moving to advanced metadata management and this is the case of MX. And there's certain  experiments who can also go into that area where the technique is so well defined that often you  can imagine doing this, doing much more than you're doing for everyone. And for this there's a talk  tomorrow which is by Gordon, Gordon Leonard who will be talking on this at this workshop and the  word there is often called I spy B. I spy B is just the technology or the implementation. Gordon  will talk about the metadata needs as well. So I want to talk about MX. What we've done has been  built up all entirely on existing standards so we didn't want to reinvent anything. We haven't  reinvented Nexus nor HDF5. We took an existing metadata catalog from SDFC. We have I spy B and  of course where things overlap with the SIF format we will use that as well. This slide  summarizes everything that we've done and how it works. So if you look over here it's not very  let's say the writing is a bit small so you'll have to read the slides afterwards.  But basically we constrain on a site-wide basis the definition of metadata. Then we link these  definitions of course we add definitions when they're missing. We link these definitions to what's  on the beam line or in the environment. Then during the experiment this metadata is collected  and finally it's ingested into an HDF5 file which is given to the user, the master file I'll come  back to that. Eventually it'll go into an e-log book. This is not done yet and then goes into the  metadata catalog as well. So it's all this approach can be applied to all beam lines.  You can have also the algorithms as well, hard works, the flowchart is basically we install  some software on the beam line. We ask the scientists what metadata do they need. We show them what's  available already, what we've already got predefined. We add the missing information, put it back in  the global definition so it's available for everyone and then we configure this on the beam line.  This is done with a number of tools with graphical user interfaces. Over here you see everything  that's defined in iCat. All of the nexus classes over here and you drag and drop the nexus classes  into the global definition and then link that to where it's stored in the database. This using a  tool like this and a global configuration file is everyone starts off from the same point and  secondly you're constraining the user not to invent things that don't exist already.  If you just zoom on this you should be able to read all of these in the slides and you can see  how this has been used on the real example at the ESRF. Today we have a number of  global definitions and we're extending this for specific techniques. This is microbeam  radiotherapy, tomography, scanning, measurement, generic scanning. This is really only the beginning  and this will be extended. The idea is to extend this to all the beam lines. Once you get to the  beam line you configure it, you configure the beam line based on what is defined in the configuration  file of what metadata are already defined for the site and then during the experiment this is  recorded and sent to an HDFI file as well as to iCat and at any time afterwards you can regenerate  this file if someone needs it. One thing that's key to this whole thing and that's where we  differ with when I see people talking a lot about image formats or that, the headers in image files  are not sufficient at all. There's a lot of the context of what makes a data set and if you look  at other techniques they have much more varied ways of collecting data. They do multiple scans,  sub-scans, in the scan they do calibration runs and things like that. All of this contextual  information cannot be put in a single image header file so you need a global file to do that and  there the scan is really the key concept that defines what is a data set. It also has to be very  flexible as well. You have to be able to mix multiple techniques in the same file because  people are inventing new techniques, the techniques which exist only on one beam line and I don't  think we can expect international standards on those. We're going to have to be flexible in accepting  and defining groups of data which don't exist elsewhere. This is for instance the input we got  from the nano beam line where we're working on holography tomography where they take multiple  tomography data sets at four distances and then they combine that together. The guy says the  scientist wants this information that's on yeah once this as metadata. I think this is very specific  to the way they do a holography and you can't expect this to be now part of some standard or  anything but you have to put this in the file and you have to make this also available. So this is  the kind of data that we're dealing with and where I think this kind of flexibility will be required  on many beam lines and on all sites as well. If you look at tomography what we've got here is these  things again are all the information that's required for the data analysis and very specific to these  data analysis programs. So nothing that can be that would be hard to make an international standard  out of that. If you look at what they're doing in radiotherapy they don't have data analysis  programs they don't analyze the data they have patients and they have to keep a protocol of  certain parameters. These have to also be stored as metadata. This can all be captured with the  system. It's a generic scan which allows us to treat any kind of multi-axis scan.  We can measure include dynamic data as well because metadata was initially defined as static  data during the experiment but there's a lot of dynamic data which needs to be captured as well.  This is captured in the measurement. So once we get to the beam line there's a lot of shovel work  to do to make this hook up everything that we've got. Here you can see the setup for on the right  hand side no left hand side. This is what's being defined on the beam line. So it's the  scientist said from this technique I need this and this metadata out of the global configuration  and on the left hand side you get the this is the description of the beam line hardware.  And by dragging and dropping elements across you link up the definition with where the source the  data is and then during runtime it just gets a trigger and says start of scan take metadata  and this goes into the file and you get a list of entries in an hdfi file and if you drill down  in those entries you find you'll find all of your metadata. Our current status is we have this  running on two beam lines and work is started started on two more. We're addressing hollow  tomography, nano fluorescence, nano diffraction and these techniques and the next beam lines want  to do spectroscopy, full field diffraction. I mean the list of techniques I can see is going to be  in the hundreds by the end of this something which is going much beyond what just talking about a  single technique. We have estimated based on this how much metadata we think we're going to generate  at the ESRF. At the moment our estimate in terms of size of the database based is a rough estimate  because we're going from a single experiment oh let's say six months of data collected on one  beam line all the time. We come up with 120 gigabytes per year for the all the experiments at the  ESRF which is a very modest figure and there's no financial reason why we can't store that forever  and this has triggered the discussion at the ESRF of the metadata policy. This has always been a  very painful still is very painful and hasn't come we haven't come to a conclusion yet but now that  we got a handle on all the metadata at the ESRF there's no reason not to actually now come up  with this metadata policy which will say which metadata are available for everyone and so we're  going to have to take this injection we're going to have to work on this and come up with a site  wide policy which has been done at ISIS has been done at ILL but to my knowledge only Elettra I don't  know if other synchrotrons have one but we're going to there's a pledge by the directors to do this  during the next during phase two that there will be a data policy for all data. The metadata is  definitely the key to any kind of improvements we want to offer any kind of services we want to  offer because none of these things can be automated if we don't have the metadata and and I think  the challenge is collecting the metadata but also doing the calibration as well this has to be done  before the experiment or just I mean very close to when you want to analyze the data so all of these  kind of extensions to data services will depend on having metadata and we in a way we block today  because we can't offer we can only offer it on an individual basis but not on a site wide basis  one of the services we were proposing to offer was called data analysis as a service and we even  prepared an EU proposal for this for H2020 but we were not successful but just to show you the key  role that metadata plays in this is you have all of this cloud structure here which allows you to  access and do offer services but everything goes via the metadata catalog. The metadata catalog is  what will identify who has access to what data and what what does the data mean so without that we  can't offer the service as well and finally how can this work drop help us I think any kind of  metadata schemes which are proposed here we'll make sure that we can somehow we'll put them in our  system to see that we can use them or see where we can find common ground maybe a metadata policy  kind of template or proposal from the IUCR would help sometimes we end up having discussions of  should the title of the proposal be public I mean things like that if international organizations  say these things are standard then maybe it'll help sites to come up with to get over these  these blocking points sometimes I think it would be great if a board like this a working group like  this said which must curate raw data and not say you can do it if you want and so on be a bit more  prescriptive on this and of course anything that's related to new techniques that are in  crystallography because I have the impression yo is very focused on the current techniques in MX  but if I think I see what people are doing with zero crystallography with meshing and things like  that are these techniques also being covered by the the current definitions I think this I mean  there's going to be a lot of evolution I think especially with the new sources we're not the  only ones doing this I think Daisy I know Daisy's doing something similar NSLS2 is also doing that  you heard from Herb similar thing I really think the key to managing the metadata is having a  master file where all the metadata is put together with links to the raw data then so it's not  only dependent on what's coming out of the detector but everything around it we hope we're  going to know in 20 years time I'll be able to well I'll probably be retired but then someone else  will be able to say we've solved this problem and I just think we'll probably be a challenge getting  all the scientists a majority of scientists to accept what we propose them in terms of definitions  because there's always people I mean things are when you start going into the details it is much more  complicated than just what a computer scientist thinks the problem is just the definition of energy  in some experiments it can be many definitions okay and the credits are most of this work is the work  of my colleagues I'm just presenting it and I'm very grateful also to have access to these photographs  which I find were quite pretty okay that was it  was faster than that was so with your idea of basically for every experiment having all the  metadata in one HDF file is then to take this one step further the idea also to include all the data  in HDF files or will they still be stored in whatever way they come out of the detectors the goal is  to store the file the data the raw data in HDF files that we can use the linking mechanism from the  master file but as you know from the SRF we have some very old data formats we have a number of  different data formats those files can still be listed in the master file but you won't be able to use  HDF file linking with it the concept that we've taken is we putting the system in place in parallel  with the existing system so it means that if people and data analysis programs can adapt when they  ready and you'll have both systems running in parallel so you'll have the old files and the old way  of storing metadata or not storing metadata plus in parallel the new way thank you  Camille Dubek, Lance Florence just a brief comment about this electronic logbook you were talking about  and the metadata which are actually not in the system which need to be introduced by the users or the  operator they're better and worse organized people and maybe it's worth of making such a kind of  table format for that with a checklist just to be sure that in the time of experiments all the needed  metadata are stored because otherwise when the experiment is finished they may be lost somewhere  lost in the papers and I agree I'm sometimes a bit I find it's exasperating how much how much  sloppiness is acceptable you know in a way it seems strange if you with other systems let's say more with older  sciences or that things are often much more organized you wouldn't imagine not starting experiment with  having put in all of that information but at least the synchrotron I think it's still a young science  and so sometimes I'm amazed at how people work here I mean it works in the end but with a lot of human effort  I sense that the ESRF senior management is taking a close interest they will have asked you the question  what will success look like what was your answer?  Success would be being able to provide new services that we don't have today around data  and that for all beam lines because at the moment it depends very much on which technique you're talking about  and the people involved so if you want to offer a cloud-based service or that or you want to offer archiving  or you want to offer automatic processing these services can only be offered if you have the correct metadata  so if we if we doing those things which we're not doing today for all techniques it means that we've actually we have succeeded  I imagined you might have answered avoiding failure  I'd be very I'm it's not so easy to define failure because if I define failure as a situation today  it would be a way of saying the synchrotron is not working and it is working and science is coming out of it  you can work like that as well  but the neutron sources would say every neutron ends up supporting a publication  Vladik gave his documentation that 99.7% of synchrotron X-rays for MX do not support a publication  so between these two extremes we have here a consistent at least metric for what success would look like  which is that with the better metadata capture and we also insisted on raw data but it was softened  we would have a much more efficient usage of the photons or neutrons that are produced  I would add one comment which was the most the biggest sort of sign of success was when the scientists  the beam line on which this has been deployed said it's now become a pleasure to do experiments  this is that meant that we had added value to what he was doing  very good  Simon Cole Southampton  so you essentially approach people a pick list of metadata items you've got the whole catalogue  and they can generate a shopping list for their particular technique  once you've done that many times do you think you'd be able to pick out what are the core terms  would you be able to come up with a core schema  that would be very useful for people outside of the central facilities world  you live in a very constrained environment in central facilities when it goes out into institutions things get messy  really messy you think you've got a messy problem  this is what happens in universities  and so if we had something that we could go on that would be very useful  I think there's nothing I mean this is all open we can share this information
Towardsageneralised_0_863.png Towardsageneralised_864_2027.png Towardsageneralised_2028_2939.png Towardsageneralised_2940_3719.png Towardsageneralised_3720_4199.png Towardsageneralised_4200_4823.png Towardsageneralised_4824_6095.png Towardsageneralised_6096_7283.png Towardsageneralised_7284_12335.png Towardsageneralised_12336_13199.png Towardsageneralised_13200_14063.png Towardsageneralised_14064_14639.png Towardsageneralised_14640_15035.png Towardsageneralised_15036_15587.png Towardsageneralised_15588_16919.png Towardsageneralised_16920_17543.png Towardsageneralised_17544_18323.png Towardsageneralised_18324_18599.png Towardsageneralised_18600_20819.png Towardsageneralised_20820_20999.png Towardsageneralised_21000_21923.png Towardsageneralised_21924_22547.png Towardsageneralised_22548_23915.png Towardsageneralised_23916_24059.png Towardsageneralised_24060_24491.png Towardsageneralised_24492_24971.png Towardsageneralised_24972_25103.png Towardsageneralised_25104_25283.png Towardsageneralised_25284_25955.png Towardsageneralised_25956_26807.png Towardsageneralised_26808_26891.png Towardsageneralised_26892_27503.png Towardsageneralised_27504_28067.png Towardsageneralised_28068_28115.png Towardsageneralised_28116_29051.png Towardsageneralised_29052_29363.png Towardsageneralised_29364_29963.png Towardsageneralised_29964_30071.png Towardsageneralised_30072_32027.png Towardsageneralised_32028_32279.png Towardsageneralised_32280_43482.png