Overview of metadata and raw data cataloguing at Diamond
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Aller, Pierre
Diamond Light Source as a relatively new facility has been able to capture and catalogue all its raw data (now over 3.6 Petabytes). Additionally as much metadata and processed data as possible has always been collated with the raw data and captured into both databases (ISPyB) for query and quick access, and into the raw data files (imgCIF/CBF and NeXus). Progress and status on these developments will be presented.
 Good morning everyone. Normally this talk should have been done by Alan, an ashtray who is managing the diamond that analyzes group and software group. So I will try to give this overview for Onis B half. So we start with the data archive. Like Brian said earlier, we generated a diamond more than 3.6 petabyte of data. We reached 1 petabyte of data in July 2013. So it was for the diffraction pattern of an RNS for the crystal diffraction at 1.6 angstrom. And at the beginning of the year, we reached 1 petabyte of data, 3 petabyte of data, sorry, with a micro CT scan of a tropical base. It was an image to understand how those bees navigate in the dense forest. Okay, so when you are a user and you want to access to your archive data, you can go through iCAD as Brian said earlier. So it's quite easy from diamond. So you just go on the website on icad.diamond.ac.uk with your FedID and your password. You can just see all the visit you have been assigned before and you just click and choose what data or data set you want to download. So we have ongoing work to improve the reliability, usability, scalability, and especially the downloading the data. So we will move to Globus to make things faster. It's just an overview of how the data is captured from every stage of the process. So first of all, the only manual entry will be done in the user office by the staff. So they will just do the generation of time and user for the session. This will update iCAD, a diamond, which contains only the session and user information. So it will update iCAD, the information catalog, and it's also connected to iSPYB and GDA. GDA, this is a diamond, the data acquisition software diamond. So all the data will be registered as Brian said on storage D and then on tape on the STFC site. So we'll move on on the LIMPS and experiment information management. So this is just an overview for the MX users. So first of all, the users that will be granted, they will request for time, like, yes, with our user office database. So the user office will grant time to the users. The user will have to, of course, comply for all the safety information and everything. And they will update their simple information in iSPYB. And the user after will have also to collect their data, of course, and everything will be recorded for their data, for the auto processing, for the fluorescence experiments they were doing and the strategy, and also also information of the beam line, detector and sample changer as well. And everything should be accessible to the user during the beam time and after the beam time through a web interface. So we have different interfaces for iSPYB. So just the website, the regular website where you can have a quick summary of what you've done about the beam line, a crystal image, and some automatic processing data. It's also available through Dawn and through an iOS app, a syncing which is available as well. So one for Android is currently under development. This, we are trying to do the same for other beam line, especially the same workflow for the tomography, for the EM facility, which is coming live very soon, and for the powder diffraction. So this is a workflow in the case for the long duration experiment on a high-level beam line. So the same thing to use with a user office for scheduling the time and we will record everything for their experiment. Or in this case, it will be for some calibration and integration for the data. And we'll have the same kind of interface for everybody. So for the powder diffractions, we'll have this kind of interface with some information about the beam line, detector and everything. For tomography, it will be the same kind of information. Here, picture, this is a pinhead. And for the EM facilities, they will have also access to all their data, the results through the iSPYB. Okay, so now I will talk about the raw data and the process data try. So we are, we are committed to move to the Nexus for as a standard. So we, the key driver are both scientific and technical. Combination, so the software, the data acquisition software, GDA and the software on the beam line for the control software, APICS, on all the beam lines, they are allowing us already to collect data in Nexus format. So we are also at the amount in striving for the HDI5 improvement. We are funding with some partners for the improvement of the HDI5 swimmer and also the virtual data set, ability to link together multiple data source files. And we are also contributed to H5Py and Java data for binding to allow user of swimmer. Okay, so for the Nexus for experiments, so we will try to have Nexus for all the data diamond. So we have a NXMX for MX, it's a Nexus format. So it's a work with Andreas and Herbert Bernstein as well. We are not really using Nexus right now for MX. It's used for another beam line on non-MX beam line. And I think we'll move really to Nexus when we'll get an IGL on a for the MX beam line. We are also using NX for Nexus for tomography and the beam line RPS, the NX-tomo and NXRPS. It's also used for the non-crystalline diffraction and soon for the PDF experiment. And we are currently investigating for NX6M and NX3O as well. So for the future integration, we are continuing to move towards a full Nexus defined raw data and process data. We will record more experiment and experiment types in LIMPS and we will have more information from the raw data file and from LIMPS to enrich our data type. And as Susanna said earlier, also we try to better link into external database and repository such as the PDB and the CDC and other facility catalogs. So it's in summary. The LIMPS system and a diamond will recall all aspects of experiment except the raw data and it will eventually cover all the experiments towards all the beam lines. The files diamond experiment produce should all eventually adhere to a descriptive standard. So for the Nexus and diamond data archive should provide a link between the data and reasonable experiment parameter results. So I would like to thank Sander Pierre for, of course, Alan Nyak, HDF5, I spy big collaborator, STFC and ICAD collaborator and diamond computing and software and science course. Thank you for your attention. Thank you very much Pierre. Questions to Pierre. Do you have a policy for making data open for the public so after a certain period? No, we won't make the data public. It's, no, in a, yeah, not channel. So it's, ICs are data is public but not a diamond. It's not something we are going to do. Not unfortunately. So basically, if I'm interested in particular experiment, which was done, my way is to contact the experiment and ask for a data. Yes. And he cannot say that he cannot, that he does not have this data because this data is in diamond. Yes. No, I mean, it's great. No, it's great. I might comment. I think there is a terminology thing here which just depend down. I think Brian used the word archive for the diamond data. I think when Alan and I discussed that before, it's not actually an archive in the way that one would formally understand the word archive. It is a preservation of data. But I think an archive in addiction definition means that data sets are accessible, findable and so on. And that's not the case. It's a preserved set of data. And certainly, the archive also means that it can be, the data can be located instantly. Its access is quite controlled, especially to the visual users. And that takes time. That's possible. But yeah, diamond's data policy says that they will comply to essentially the user's data policy. The user may well be subject to data policies like research councils data policies. So it may be made, have been made open by the user, but diamond aren't going to say that. That's up to the user to decide that. Yes. Yes. Yes. That is how diamond are applying the policy. It's not the same way as eyes is applying. Eyes is taking a different point of view. There is a difference in point of view. On slightly tangential point, I was intrigued to see that you've developed iOS Android interfaces. What do they do and how are people using them and how many people are using them? I don't really know the number of people using it actually. But I think the website is working very well for iSpy beats working very well as well on mobile phone and everything. But I think Michael can help us. Please wait. Could you repeat the answer just for the all? Maybe if you get to Michael. The website provides a way of accessing metadata. So in terms of the actual images collected, the angles collected in a set of images, it also provides a way to kick off data processing tasks using the cluster of diamond. That data is then reprocessed. You can go and have a cup of coffee or something and then come back and get your data back to your home desktop to reprocess. So I'm not actually using a mobile device to capture metadata. I mean it's quite a handy thing to interact with and that's the most likely option of getting a lot of manual metadata. We're not actually capturing the metadata by mobile device. No, we're purely using the mobile device currently to provide just another mode of accessing the database. Yeah, it's an interesting idea and I was going to come and talk to you more about this kind of thing later on anyway. And somewhat related question. Let's say that somebody collected 20 datasets on the same type of crystal. Do you know which one was used and only one dataset was used for structure determination? Let's say with Liga. Do you know which one? Which one would be used for what? For structure determination of Liga, let's say, of complex with Liga. Do you know which one? And if you don't and if this information is not somewhere, I can bet with you that after two years the person who supervised the PI, who supervised the experiment, does not know, maybe his postdoc knows, but his postdoc will move somewhere else and if somebody will ask him what has happened, he will answer, look, I have a lot of present projects and I really do not have time to look into that. So my question is, and this is the real question, are we going to the way of genomics? Yes, not genomics, but sequencing. Yes, when now the old sequence data cannot be even put somewhere because they are completely disorganized and it's simpler to do that again than to find something in the archive. It's because we do not have archives. We have something which is the backup for users and this backup should be very short term if it will be in the present form. If it won, there is really no metadata in there. Yes, there are data, but not metadata. I think you should be able to retrieve your data, to know which dataset has been used to solve your structure and who is the ligand. In ISPI-B, it's quite obvious or it should be obvious enough for you to find which dataset has been used. Yes, at least. No, ISPI-B, no, it would be available for a long life. Do you want to add something? Just to follow up on that, you're quite right with in macromilacocrystallography we're leading the world really in terms of collecting the data, but in terms of physical sciences, I think there's still quite a long distance to go. This is exactly what we're starting to do now with rolling out ISPI-B for other fields such as power diffraction, such as PDF and also for electron microscopy. The Nexus file for this is a real help for this in terms of actually being able to get the data or structure metadata in terms of the entire beam line layout and such. But then there's also, there's still plenty of wiggle room within the Nexus specification where we can put the kind of information that you're asking for of what has this data been used for subsequently. I think the Nexus format is certainly at diamond, this could be a very good solution to some of the problems that we're talking about here. Right on that, thank you very much indeed again Pierre and very, very clear, very comprehensive, thank you. So it's a great pleasure to introduce Brian McMahn.
 Good morning everyone. Normally this talk should have been done by Alan,  an ashtray who is managing the diamond that analyzes group and software group.  So I will try to give this overview for Onis B half. So we start with the data archive.  Like Brian said earlier, we generated a diamond more than 3.6 petabyte of data.  We reached 1 petabyte of data in July 2013. So it was for the diffraction pattern of an  RNS for the crystal diffraction at 1.6 angstrom. And at the beginning of the year,  we reached 1 petabyte of data, 3 petabyte of data, sorry, with a micro CT scan of a  tropical base. It was an image to understand how those bees navigate in the dense forest.  Okay, so when you are a user and you want to access to your archive data, you can go through  iCAD as Brian said earlier. So it's quite easy from diamond. So you just go on the website  on icad.diamond.ac.uk with your FedID and your password. You can just see all the  visit you have been assigned before and you just click and choose what data or data set you want  to download. So we have ongoing work to improve the reliability, usability, scalability, and  especially the downloading the data. So we will move to Globus to make things faster.  It's just an overview of how the data is captured from every stage of the process.  So first of all, the only manual entry will be done in the user office  by the staff. So they will just do the generation of time and user for the session.  This will update iCAD, a diamond, which contains only the session and user information.  So it will update iCAD, the information catalog, and it's also connected to iSPYB and GDA. GDA,  this is a diamond, the data acquisition software diamond. So all the data will be registered  as Brian said on storage D and then on tape on the STFC site. So we'll move on on the  LIMPS and experiment information management. So this is just an overview for the MX users.  So first of all, the users that will be granted, they will request for time, like, yes, with our  user office database. So the user office will grant time to the users. The user will have to,  of course, comply for all the safety information and everything. And they will update their  simple information in iSPYB. And the user after will have also to collect their data, of course,  and everything will be recorded for their data, for the auto processing, for the fluorescence  experiments they were doing and the strategy, and also also information of the beam line,  detector and sample changer as well. And everything should be accessible to the user  during the beam time and after the beam time through a web interface. So we have different  interfaces for iSPYB. So just the website, the regular website where you can have a quick  summary of what you've done about the beam line, a crystal image, and some automatic  processing data. It's also available through Dawn and through an iOS app, a syncing which is  available as well. So one for Android is currently under development. This, we are trying to do the  same for other beam line, especially the same workflow for the tomography, for the EM facility,  which is coming live very soon, and for the powder diffraction. So this is a workflow in the case for  the long duration experiment on a high-level beam line. So the same thing to use with a user  office for scheduling the time and we will record everything for their experiment. Or in this case,  it will be for some calibration and integration for the data. And we'll have the same kind of  interface for everybody. So for the powder diffractions, we'll have this kind of interface with  some information about the beam line, detector and everything. For tomography, it will be the same  kind of information. Here, picture, this is a pinhead. And for the EM facilities, they will have  also access to all their data, the results through the iSPYB. Okay, so now I will talk about the raw  data and the process data try. So we are, we are committed to move to the Nexus for as a standard.  So we, the key driver are both scientific and technical. Combination, so the software, the data  acquisition software, GDA and the software on the beam line for the control software, APICS,  on all the beam lines, they are allowing us already to collect data in Nexus format.  So we are also at the amount in striving for the HDI5 improvement. We are funding with some  partners for the improvement of the HDI5 swimmer and also the virtual data set, ability to link  together multiple data source files. And we are also contributed to H5Py and Java data for binding  to allow user of swimmer. Okay, so for the Nexus for experiments, so we will try to have Nexus for  all the data diamond. So we have a NXMX for MX, it's a Nexus format. So it's a work with Andreas  and Herbert Bernstein as well. We are not really using Nexus right now for MX. It's used for  another beam line on non-MX beam line. And I think we'll move really to Nexus when we'll get an  IGL on a for the MX beam line. We are also using NX for Nexus for tomography and the beam line  RPS, the NX-tomo and NXRPS. It's also used for the non-crystalline diffraction and soon for the  PDF experiment. And we are currently investigating for NX6M and NX3O as well. So for the future  integration, we are continuing to move towards a full Nexus defined raw data and process data.  We will record more experiment and experiment types in LIMPS and we will have more information  from the raw data file and from LIMPS to enrich our data type. And as Susanna said earlier,  also we try to better link into external database and repository such as the PDB and the CDC  and other facility catalogs. So it's in summary. The LIMPS system and a  diamond will recall all aspects of experiment except the raw data and it will eventually cover  all the experiments towards all the beam lines. The files diamond experiment produce should  all eventually adhere to a descriptive standard. So for the Nexus and diamond data archive should  provide a link between the data and reasonable experiment parameter results. So I would like  to thank Sander Pierre for, of course, Alan Nyak, HDF5, I spy big collaborator, STFC and  ICAD collaborator and diamond computing and software and science course. Thank you for your attention.  Thank you very much Pierre. Questions to Pierre.  Do you have a policy for making data open for the public so after a certain period?  No, we won't make the data public. It's, no, in a, yeah, not channel. So it's,  ICs are data is public but not a diamond. It's not something we are going to do. Not unfortunately.  So basically, if I'm interested in particular experiment, which was done, my way is to contact  the experiment and ask for a data. Yes. And he cannot say that he cannot, that he does not  have this data because this data is in diamond. Yes. No, I mean, it's great. No, it's great.  I might comment. I think there is a terminology thing here which just depend down. I think Brian  used the word archive for the diamond data. I think when Alan and I discussed that before,  it's not actually an archive in the way that one would formally understand the word archive. It is  a preservation of data. But I think an archive in addiction definition means that data sets are  accessible, findable and so on. And that's not the case. It's a preserved set of data.  And certainly, the archive also means that it can be, the data can be located instantly.  Its access is quite controlled, especially to the visual users.  And that takes time. That's possible. But yeah, diamond's data policy  says that they will comply to essentially the user's data policy. The user may well be  subject to data policies like research councils data policies. So it may be made,  have been made open by the user, but diamond aren't going to say that. That's up to the user to decide  that. Yes. Yes.  Yes.  That is how diamond are applying the policy. It's not the same way as eyes is applying.  Eyes is taking a different point of view. There is a difference in point of view.  On slightly tangential point, I was intrigued to see that you've developed iOS Android interfaces.  What do they do and how are people using them and how many people are using them?  I don't really know the number of people using it actually. But I think the website is working  very well for iSpy beats working very well as well on mobile phone and everything.  But I think Michael can help us.  Please wait. Could you repeat the answer just for the  all? Maybe if you get to Michael.  The website provides a way of accessing metadata. So in terms of the actual images  collected, the angles collected in a set of images, it also provides a way to kick off  data processing tasks using the cluster of diamond. That data is then reprocessed. You can go and  have a cup of coffee or something and then come back and get your data back to your home desktop  to reprocess.  So I'm not actually using a mobile device to capture metadata. I mean it's quite a handy  thing to interact with and that's the most likely option of getting a lot of manual metadata.  We're not actually capturing the metadata by mobile device. No, we're purely using the  mobile device currently to provide just another mode of accessing the database.  Yeah, it's an interesting idea and I was going to come and talk to you more about this kind of  thing later on anyway. And somewhat related question. Let's say that somebody collected 20  datasets on the same type of crystal. Do you know which one was used and only one dataset was used  for structure determination? Let's say with Liga. Do you know which one? Which one would be used for  what? For structure determination of Liga, let's say, of complex with Liga. Do you know which one?  And if you don't and if this information is not somewhere, I can bet with you that after two  years the person who supervised the PI, who supervised the experiment, does not know,  maybe his postdoc knows, but his postdoc will move somewhere else and if somebody will ask him  what has happened, he will answer, look, I have a lot of present projects and I really do not  have time to look into that. So my question is, and this is the real question, are we going to the  way of genomics? Yes, not genomics, but sequencing. Yes, when now the old sequence data cannot be even  put somewhere because they are completely disorganized and it's simpler to do that again than to  find something in the archive. It's because we do not have archives. We have something  which is the backup for users and this backup should be very short term if it will be in the  present form. If it won, there is really no metadata in there. Yes, there are data, but not metadata.  I think you should be able to retrieve your data, to know which dataset has been used to solve your  structure and who is the ligand. In ISPI-B, it's quite obvious or it should be obvious enough for you  to find which dataset has been used. Yes, at least. No, ISPI-B, no, it would be available for a long  life. Do you want to add something? Just to follow up on that, you're quite right with in macromilacocrystallography  we're leading the world really in terms of collecting the data, but in terms of physical  sciences, I think there's still quite a long distance to go. This is exactly what we're  starting to do now with rolling out ISPI-B for other fields such as power diffraction,  such as PDF and also for electron microscopy. The Nexus file for this is a real help for this  in terms of actually being able to get the data or structure metadata in terms of the  entire beam line layout and such. But then there's also, there's still plenty of wiggle room within  the Nexus specification where we can put the kind of information that you're asking for  of what has this data been used for subsequently. I think the Nexus format is certainly at diamond,  this could be a very good solution to some of the problems that we're talking about here.  Right on that, thank you very much indeed again Pierre and very, very clear, very comprehensive,  thank you. So it's a great pleasure to introduce Brian McMahn.
OverviewofmetadataandrawdatacataloguingatDiamond_0_455.png OverviewofmetadataandrawdatacataloguingatDiamond_456_503.png OverviewofmetadataandrawdatacataloguingatDiamond_504_1523.png OverviewofmetadataandrawdatacataloguingatDiamond_1524_2603.png OverviewofmetadataandrawdatacataloguingatDiamond_2604_4019.png OverviewofmetadataandrawdatacataloguingatDiamond_4020_4163.png OverviewofmetadataandrawdatacataloguingatDiamond_4164_5759.png OverviewofmetadataandrawdatacataloguingatDiamond_5760_6191.png OverviewofmetadataandrawdatacataloguingatDiamond_6192_6299.png OverviewofmetadataandrawdatacataloguingatDiamond_6300_6551.png OverviewofmetadataandrawdatacataloguingatDiamond_6552_6695.png OverviewofmetadataandrawdatacataloguingatDiamond_6696_7559.png OverviewofmetadataandrawdatacataloguingatDiamond_7560_8279.png OverviewofmetadataandrawdatacataloguingatDiamond_8280_8411.png OverviewofmetadataandrawdatacataloguingatDiamond_8412_9143.png OverviewofmetadataandrawdatacataloguingatDiamond_9144_9863.png OverviewofmetadataandrawdatacataloguingatDiamond_9864_11471.png OverviewofmetadataandrawdatacataloguingatDiamond_11472_12431.png OverviewofmetadataandrawdatacataloguingatDiamond_12432_13283.png OverviewofmetadataandrawdatacataloguingatDiamond_13284_13775.png OverviewofmetadataandrawdatacataloguingatDiamond_13776_13967.png OverviewofmetadataandrawdatacataloguingatDiamond_13968_20507.png OverviewofmetadataandrawdatacataloguingatDiamond_20508_20579.png OverviewofmetadataandrawdatacataloguingatDiamond_20580_20855.png OverviewofmetadataandrawdatacataloguingatDiamond_20856_22283.png OverviewofmetadataandrawdatacataloguingatDiamond_22284_24659.png OverviewofmetadataandrawdatacataloguingatDiamond_24660_24767.png OverviewofmetadataandrawdatacataloguingatDiamond_24768_26051.png OverviewofmetadataandrawdatacataloguingatDiamond_26052_28175.png OverviewofmetadataandrawdatacataloguingatDiamond_28176_28223.png OverviewofmetadataandrawdatacataloguingatDiamond_28224_28479.png