OpenML: Open, Networked Machine Learning
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Vanschoren, Joaquin
Today, the ubiquity of the internet is allowing new, more scalable forms of scientific collaboration. Networked science uses online tools to share and organize data on a global scale so that scientists are able to build directly on each other's data and techniques, reuse them in unforeseen ways, and mine all data to search for patterns. OpenML.org is a place where researchers can easily share and reuse machine learning data sets, tools and experiments. It helps researchers win time by automating machine learning experiments as much as possible, and gain more credit for their work by making it more visible and easily reusable. Moreover, OpenML helps scientists and students to explore different machine learning techniques, find out which are most useful in their work, and collaborate with others to analyze scientific data online.
 Okay, welcome to the second talk. So I will introduce OpenML for those who are probably less familiar with it. This is the site. So the goal of OpenML is to enable network machine learning. It's basically what OpenML tries to enable is for researchers, machine learning research, data scientists, working in all sciences to be able to collaborate together on a global scale and collaborate on a very detailed level. It means that if I'm running an experiment, I have some data sets around machine learning reference on this, I can automatically share those results with the world. And somebody in China can then immediately reuse those results in their work, somebody in the US may then compare his method with my method. And the way we do this is that it all happens automatically by integrating machine learning tools so that if we use those machine learning tools, they will automatically upload all the details of my research online. Most importantly, it will link all results. If I run a machine learning study on certain data sets, it will all be automatically linked to all the other studies that have been run on the same data set. That means that I can look at, I can share particular data sets and I can challenge everybody in the world to build the best machine learning workflows on top of this and I can always check which are the best techniques. It also means I can do these large front searches like in Metacusar and share all this online and they will be automatically organized online so I can just anytime go online and see what people have been doing. It could be that someone in the world as a robot scientist there making computations or some algorithm that runs continuously machine learning experiments and I will always be able to get the up to date results. And very important that this happens mostly automated. So for data scientists that use those tools, there will be no extra work. They just have their username and password and they will be able to stream all the experiments online. They will be able to connect to other scientists, people they trust and share the results even before they are published. So I can start large cooperation with 20 people all over the world and they can see my experiments the second they are finished. You can see them online. And what this, what you have to achieve with this is to tackle much harder problems. So machine learning is becoming such a crucial tool in many sciences and many sciences are tackling very hard problems. And there is so much more we can do if we connect ourselves to this network. There is a lot of latent expertise. I may be working on a certain study but I may not know some method or I may not have resources to run some method. And by using OpenML, by connecting online, I can use this latent expertise. Somebody else in the world may have just the right expertise to help me at that time. So let me show you this works. So first of all, because many people don't really believe that we can actually change the way we do science. And if you look back at history, you can see science has dramatically changed. The way we do science has dramatically changed over time. In the time of Galileo 400 years ago when he discovered Saturn rings, he didn't write scientific paper. For him, having the knowledge that Saturn has these rings is much worth, much more to him if it's a secret than when he shares it. When he shares it to him, it's basically become worthless. He never shares it. The only thing he did is write down his discovery, then jump all the letters into an engram and he sends this engram to all his restaurant arrivals. What is insured to this is if somebody else makes the same discovery, he can go back and say, see, if you unscrambled these letters, you can see I was first so I should get the credit for that convention. This was how many people like Leigh Rondo and Avinci worked. For them, having knowledge as a secret was worth much more. This changed mostly through the work of the Royal Society. The Royal Society was a group of researchers and then most of us think nobody's worth for it. They said, you cannot just make a claim and assume that we will believe you. If you make a claim, you should come here and show the experiments and explain your methods and then only if we see it actually happening, we will believe that the claim is correct. They started journals to be able to disseminate both the findings and the methods. That meant that all the other people in the world could reuse the same methods. That meant that you didn't have to be incredibly relfal or you didn't need to be a genius to do science. You could actually look at what other people were doing and build on top of that. They implemented this method in the best way of disseminating information at times which was a printing press. No, it was a giant advance for science because science exploded. But 300 years later, you can ask yourself the question whether journals are still the best way to do this. Right now, we deal with very complex code. We deal with huge data sets. If we publish results in papers, they detach from all that code and data. So you should work on systems where this is all integrated. Also, in a paper, there's no room to put all the experimental details. If you want to reproduce results, you have to make guesses. Papers are not updatable. You have to hope that other people will cite you and that the set-up is counted correctly. There's publication bias, negative results are not published typically. There's no way to really discuss a paper online. Although recently, scientists are using Twitter to discuss scientific papers. It's very interesting. The best discussions of scientific papers happen on Twitter. Open science is still, I think we're moving in the right direction, but still a question mark whether we can change this journal industry to be more open access. And you still do this while we have this internet and there's very great tools. You can collect results to open online databases, to collaborative tools for code development. You can build web service API to easily collect data from many different places. We can use scalable algorithms to mine two incredibly large data sets. We can go online and have discussions. We can even include citizen scientists to help us if we have particular problems. And many scientists have already been thinking along those lines. One of them is, for instance, Tim Gowers. He's a field medallist, right? One of the great mathematicians of this age. And he had a hard problem in mathematics. He decided one day he was stuck on it and instead of just putting it in his drawer, he said, okay, I'm going to just put all my partial information online. And everybody in the world can look at my train of thought and can add to this and maybe we can collaboratively solve this hard problem. Within a month, many, many people have been having active discussions and within 40 days have actually solved the problem. They actually solved an even harder generalization of the original problem. And what this does is it actually combines the many minds towards solving a particular problem. And if you go, if you have to find out, I need those and those and those people to have the right expertise, that's very hard to do. But if you work openly, if you say, okay, I'm solving this problem, if you want to help, you can help me, you kind of attract the right expertise from people and by working together you can actually solve much harder problems together. And the way they do this, they make jump publications out of the proofs that they found based on who contributed what to the findings. It's also apparent in the SDSS, the Sloan Digital Sky Service. This is a robotic telescope that slowly scans the universe. And very interesting here is that after one year, they put all the data online. That means everybody in the world can just say, okay, I want to look at that part of the sky, those wavelengths, and you can get imagery from that part of the sky. This is a huge resource, this is basically one of the best resources in astrophysics. Many people have been using this to answer questions. And the idea here is that you broadcast data hoping that other people will have interesting things to discover in the data. Because when you collect data, you don't always know exactly what kind of questions you can answer. But somebody else may think, hey, if I had that data, I could answer these kind of questions much quicker. This is what the SDSS basically does. And this works incredibly well. There's thousands of papers on new discoveries, reducing this data. For instance, there were two astronomers thinking about what is possible for two black holes to overdure on each other. And instead of running very complicated and expensive simulations, they actually downloaded data, they used machine algorithms, and actually discovered this happening in space. Most interesting, while there are only about 10,000 astronomers, they have only one million people looking at the data. That means that 99% of the people looking at the data are not astronomers. They are school teachers and people, amateur astronomers. People are just interested in looking through space. And sometimes they make new discoveries. One of the discoveries happens through Galaxy Zoo. So one of the problems that they were having is that they have about a million galaxies and to analyze them, they need to classify them. This is a left-turning spiral galaxy or right-turner or it's like a beam or a ball galaxy. And instead of hiring PhD students to manually annotate all these galaxies, they made a website and say, look, everybody can go online, you get a picture of a galaxy and you can classify it. It was usually successful. I think at some point they had 70,000 classifications an hour. All the people in the world, like looking through these galaxies, it blew their servers. But this really shows that if there's interesting information, people will be interested in looking at this and especially at the rates that we collect scientific data these days, we collect data faster than we can actually look at it. So you as a citizen have actually a chance of seeing a galaxy for the first time. You can be the first human being to look at this galaxy. And it's the same happening in biology. Maybe you're the first human to look at a certain brain scan or whatever that's being shared through open science. So one of the things we discovered is you have these little green blobs sometimes. And most professional astronomers thought this was just noise. But then it was once a Dutch school teacher who said, I don't think this is noise because I've seen this many, many times. She started a blog. People started looking for this. They found many new examples of this. And this turned out to be a new kind of galaxy. And they've discovered about three new types of heavenly phenomena in this way. And they do the same thing. You can go online. You can hunt for planets around missing stars. You can look at plankton to discover new types of plankton. It's really amazing. So how come that these projects, by working more openly, make new discoveries? What is there different from doing the normal way of doing research? Well, this can be best described as this design to interpret it. It means that by broadcasting data and broadcasting questions, you can make spontaneous discoveries. By just the right person looking at this, having an idea, or having just the right expertise to interpret the data. And by doing this, you make discoveries otherwise are missed. If you are a scientist, you have a certain mindset, you may overlook things easily. But if you share the data openly, more people will be able to find those, to interpret it correctly and make new discoveries. Also, it's hard for one scientist, easy for another. I can be working on a study for, say, I'm a bioinformatician. I'm using this shielding algorithm and getting better results. I don't know why. And instead of having, if there's somebody else who, maybe the guy who designed the algorithm, or maybe somebody else who knows a lot about it, may say, oh, I know it's going wrong. You should use, first of these parameters, or you should reformat the data in some way. If you work online this way, you may be able to find the right person to actually help you in your research. Also, what's surprising for one scientist, say you have a weird result, and you probably cause it away, for another scientist, he may be able to explain exactly what's happening, or he may have a new idea that sparks new discoveries. And this network effect means that you can do many more studies that were previously impossible. If you want to give just so many things you want to do, and you have so little time, by working more openly, you can say, OK, I have this interesting data, this interesting result. I don't have the time to look at it, but I'll share it anyway. And maybe somebody else may be able to do something interesting with it. And maybe I have a problem, and I don't want to wait a year to get a grant to analyze this, but I can maybe work online with other people and do this much more quickly. By working together, these studies become much more cheaper, much more faster. Now to do this, you need the right tools, you need the right online tools to communicate, to share results, and to share at the right level of detail with each other. So it means you have to remove the friction that's currently very common in scientific research. We do experiments that we email them to somebody else and have to reformat it and look at it. And it's all very high friction environment, and this kind of inhibits this rapid interaction. You don't want to spend an half a day like putting two days together. If it's already there online, you will be much more inclined to look at it. So it means that you have to organize the data into one coherent whole. If you have data from one researcher, another, another, another, it should be easy to take all that information together. I'm not going to spend a day looking at what each of them did and then making them together. If you have an online system that does it automatically, I can easily use the results of thousands of researchers and combine them in new studies. It's also important to make microcontributions. It should be possible to upload a single data set, to do a single experiment, or to make a single comment in a scientific discussion. Because many people won't spend a day looking at something and then making a comment. They want to do it more quickly. It should have easy organized communication, so we can actually communicate in real time with each other. There should be a clear honor system that if you work with somebody else, you give them credit and the system should enforce that. The people don't just run away with your data. If they reuse it, you are being credited for this. You can do this automatically. If you use your data, you can automatically track this. It should also protect preliminary work. I want to work with, say, I'm working on a problem like finding new drugs. I don't want the whole world to see this. I need to be able to work together with a group of people that I really trust. Those people can see my latest results. Only after the papers published, or the patent is published, we made this data openly and from there on everybody can use it. The important thing is that making a public is just a push of a button. It doesn't mean I have to spend two weeks doing this stuff because I'm not paid. I'm not going to get extra credit for making this data nice to be used by other people. If it's from the start, if it's nicely organized, just a click of a button, everybody can reuse and build on this data. It also means more people will reuse your data and it also means that more people will cite you and this gives you, again, more visibility and more credit. We should find new ways to measure impact. This is why we developed OpenML. The way this works is anybody can upload data. Say your data scientist is working on a problem, say, for instance, there's data from the Metacuser project. Just upload it online. You cannot see it because it's still private. It's on there. You can say, this is my data. This is what I'm trying to do with it. This is what I'm trying to discover. Everybody in my collaboration or everybody in the world can then help me do this. So once you upload the data set to OpenML, it will first analyze the data. It will extract meta features from the data. If it's fingerprints, it may look how many features, how many columns are there, how do landmarks perform on it. If it's structured data, you may have some information about the structure of the data. If you upload data to OpenML, it will try to extract information for you. So you can always link the results to the properties of the data. It also makes a structured index where you can easily search for the data and it makes machine-readable tasks. So it says, okay, this is data. This is what I want to do with it. And you make a little XML file so that software tools that can use this can automatically interpret this. So then scientists, they can design workflows, get solved a task, and then they can upload the predictions. And the way this works, I typically have my software tool. I say, okay, run on this task. It will download the task. It will download the data sets. It will perform the data analysis. And yet it will upload the results back to the server. And then I can go online and see what's been happening. So if you upload the results, OpenML will store the full details, the full workflow, the prem settings. I know exactly what I'm doing and everybody else can see what I have been doing. So all results are easily reproducible. It analyzes the results. In some cases, like classification, you want a score, like how good is this classification and OpenML will compute these scores. And it will organize results online and visualize them. And when that's done, scientists can still go online and then reorganize the results and then they can discuss the results with each other, clean it, link it to papers, and such on. So the OpenML works. This is how it looks. So you have a number of data sets. So let me just look for data sets. Say the cover type data set. You can type in cancer or malaria and you can see what the assets exist about the data. So, say upload the data sets. So then first of all, there's the description of the data sets and it's like a wiki. So if you click the edit button, you can automatically add information. So if you know more information about data sets, everybody can edit this description and make it more complete. Then the server computes properties. So you see how many instances, how many features there are. But it also has a library of different ways to annotate these data sets. So you can see it calculates a whole list of features about the data sets so you can relate the results of your machine learning performance to this property of the data set. So in this way, this is important for instance, if you want to know which kinds of algorithms work well and which kind of data, then you can do this meta-learning studies. So you can pair these features to the performance of your algorithm and then you can build decision trees or whatever to learn relationship between the features of the data, the props of the data and the different machine learning algorithms and how well they do. And to also do some statistics about the data sets so you can see the class distribution. You can look at all the features and you can see how values are distributed there. So maybe you find something funny in there like a value that's not normalized or a feature that basically uses because it always has the same value and so on. And then below you have the tasks. So once you have a data set, people can design tasks on top of that. So one task would be classification. It means here's a data set, here's a target feature. Now build me a model that predicts the target feature. So if I click at this time, so here I see a classification, I have data streams, I have learning curves. So let's take classification. Now it's collecting all the experiments that people run on this data set for classification. So it gives description of what the classification is supposed to do. It gives you the inputs like the cover type data sets, the constellation you need to use, the target feature and so on. And these are the output sets that are expected from you, like the evaluations, the models and the predictions. And then you see, okay, so 126 runs have been computed. You can score all these runs by different evaluation metrics, depends on what you're interested in. Here you see an overview of all the different machinery methods that have been run on this data. You can see how well they performed. And each one of the dots is run, is a model that people build. It's a bit hard on iPad, but if you click on this, you get all the details of the run. Okay, right, so, okay, you can see who has run the experiment when, what's the algorithm that they used, which is the parameters that were used. You can download the full predictions, instance on instance level predictions. You can look at the model, you can actually download a serializable binary of the model. So you can actually reuse the model to make new predictions. And then you see the evaluations that the server computed on the data. So area on the rock curve is 95% and so on. It took 2600 seconds to build this model and such and such and such. So what OpenML did here is somebody has given a data set, asked the classification with it and here you see all the results by all the people who have run classification on this data set. You can do the same for stream mining algorithms. So here I can look at data stream algorithms, here I get a different task, there's a different set of algorithms that are being used to solve this task and so on. So you can define any kind of task like clustering, you can have a special meta-qsr task to look at only those results and so on. Okay, so you can search anything. Okay, so there's an, okay, I'll just switch stuff. You also have the details of the workflows, right? All the experiments that are visible are also repeatable, it means you can download the data, you can also download the code that's being used, right? So say I'm interested in running forests. So here I can see how running forests performed, again I can choose a performance metric. I can see how it performed on different data sets, that's quite a few, and I can also see the variance. So every point here is a model and the variance is caused by different parameter settings of the random forest and there's also a little help to visualize the effect of a parameter and here you see, so the red means random forest, a large number of trees, blue means random forest is smaller, you can see what the effect is of the number of trees on a performance over different data sets. You can use four parameters settings. Okay, so this is the data that's been organized. The most crucial thing is that this happens automatically, right? So people don't want to upload manually upload data, it should all happen automatically, right? And so OpenML has a web API, it can be linked to data resources like QAML, like QCi, like ML data where it can extract the data sets from, it will then make them easily available and easily searchable so that you can do large scale experiments. It also links to code repositories like tools like Weka, like also like code on GitHub, if you have code on GitHub you can also just link it to the GitHub page. It has a Java API to connect it to Java based machinery tools like Weka, Moa, Webminer, NIME. It also has an R API to link it to R packages like MLR and data mining with R or the next package that Liz will invent. So you can go online, you can look at the web API, explain quite well what you need to do to talk to the server. There's of course a Java API, in that case you just have, you create a Java object client and then you use a client to talk to the server with a certain set of methods, same with R. In a few lines of code you can actually do an experiment, download, run an algorithm, upload the results again. There's also what's being used in the QSAR project. It's also directly integrated into graphical tools like Weka. So if you use Weka, the next version of Weka will actually have it integrated. So you just select the OpenML experimenter and then for the result destination you select OpenML, then you can give a number of tasks, you get a number of algorithms and then Weka will run all the algorithms and all the tasks, automatically upload all results to the server. Same with Mova, you can select tasks, you can select algorithms and automatically all results will be uploaded. Online, say Webminer, you can have an operator, it's called Node, that actually downloads the data, then you can build your workflow at the end, you have a separate operator that again uploads all results online. So what this means is that all the people that do data science research can use Weka, Webminer, R and with a little extra effort to talk to the server, they can automatically share all their results online. Let me say everybody else can reuse their results. You can set up large collaborations and get second by second updates on what everybody is doing and all the information is shared online. You can build on top of each other's work. What are the benefits for scientists? Why would they do that? First of all, you gain a tremendous amount of time. There's always a tail of scientists telling, well, 10 years ago when I was a young researcher, we had to do this and this and this manually. Hopefully in the future, all of these things will be automatic. It means finding data and code should be just like a search query and you can see all the interesting data sets, all the interesting code. You can run that against that and have all the experiments automated online. That means you can run large scale studies. I can do huge parameter sweeps and have all results organized online and just doing queries and I can build models. I can build better models on top of that without having to do all the stuff manually. We used to do this stuff in Excel. Nowadays we can do it much, much better. We can immediately compare to the state of the art. I don't have to email other authors, so please give me your code so I can compare against it, now I can just compare against everybody who shared the results. I can immediately compare against them without actually running their code. I can just reuse their results and if I'm skeptical, I can always rerun the code because the code is available and the data is available. So it gives me a lot of time and I can do many more studies that were impossible before. It leads to more productivity because I can directly work, I can build directly in prior work. I can look at an interesting paper and I think, okay, I'm interested in this. I want to do more experiments with this. Let's download the data and let's do interesting stuff with it. This wasn't possible before. You needed weeks, maybe months to reproduce the same experiments. You can work in online teams. If you need expertise, you can invite other people to look at this data and you don't have to email them data. They can just go online. You just say, okay, add this person to the collaboration and they can look at all your data, visualize it whatever they want. You can quite source problems. So if you have a problem, for instance, you have this data that you need to process in some way. You don't have the right expertise. You can always call people to help you. You can quite source all these problems. Overall, it gives you more visibility, right? Because especially outside the sciences, because if you talk to people with industry or the general public, they will not read scientific papers. Most of them won't read scientific papers. But here you can just go online and look at all the algorithms and you can see how well they performed. So it's much easier for people, especially for students to see what exists, what's the state of the art, who are the best people in this area. And we can actually rank people with leaderboards. It also gives you more reputation because by sharing your data, other people can more easily reuse it, which leads to more certations. OpenMell can actually tell you if you reuse data how they should cite you. You can ask somebody, we use my data, this is how they should cite me. And OpenMell can also automatically track if they reuse results in a different study. OpenMell can see, okay, this study contains experiments from these 100 people maybe. Right? You can track that. And also by sharing results, you build trust, right? If you publish a paper and the reviewer can actually see all the data and all the algorithms this gives the reviewer all the more reason to accept the paper because clearly you have done all your work correctly. For visibility, OpenMell is used all over the world. So this is the statistics for August. So the big group, this is just, no. This is me and Jan working on it. But I mean, people all over the world are at least checking what information is in OpenMell and what data they can reuse. Who said the Far Far East? Yeah. Even on Hawaii. This is interesting. These little islands. I don't think you was here actually. Yeah. Okay. You have to work on Greenland though. Okay. There's also huge benefits for students. If you're a PhD student, you can easily get an overview of the state of the art. If you want to get noticed, you can participate in these large scientific problems and show what you're worth. You can get in touch with the big scientific minds of your age. You can ask questions and students can help each other to answer questions like, like happens on Quora, but here's like specific for machine learning. Okay. Things that will appear soon, I hope at the end of this week, will be OpenMell studies so that you can just say, okay, I want to have all results on this, on this data sets and this code. I build a study on this. I write a description of it. And if I publish a study, I can just put a link in the paper that links back to the special page where all my experiments are and everybody can look at that. I also want to reorganize data by tagging. So say I want to do a support factory machine study. I can tag the different algorithms with support factory machine and I can then build a study out of that. I also want to work in teams. So it's already possible to set up a private team of people to work with, but we want to have more support for that so you can easily add more people to the collaboration. Right? We also want to work on reputation building. So if you share a data set or you share an algorithm, you can see how many people are looking at it, how many people are reusing it and such. And what we definitely want to go to in the next, not this week, but in the next couple of years maybe, is to go towards kind of a robot scientist for machine learning. So in a situation where you upload a data set and there will be clever algorithms that say, okay, these are the most interesting machine learning methods to try and it will use algorithm configuration techniques to fine tune those methods to work best on the data. And you can do this because if you share all this information about different workflows, different machine approaches, different program settings, you can build metamodels that actually predict based on the features of the data which methods may be more useful. And that way, I hope it will be much easier for scientists to get help from the system. You can upload a data set, you can ask the system, give me your best guess what's the best method you can do against this. And you can actually do like a challenge, right? You can actually put humans against the machine, where the machine says, okay, I want to use this workflow and this is the best result I get and you must try to do better than the machine. It will be very interesting. Okay, that's about it. I would say you are already part of the club, but feel free to join. This isn't very old sort of pictures. We should take new pictures. And yeah, I look forward to a great week of working with you and if you have questions, just shoot. Thanks. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
 Okay, welcome to the second talk.  So I will introduce OpenML for those who are probably less familiar with it.  This is the site.  So the goal of OpenML is to enable network machine learning.  It's basically what OpenML tries to enable is for researchers, machine learning research,  data scientists, working in all sciences to be able to collaborate together on a global  scale and collaborate on a very detailed level.  It means that if I'm running an experiment, I have some data sets around machine learning  reference on this, I can automatically share those results with the world.  And somebody in China can then immediately reuse those results in their work, somebody  in the US may then compare his method with my method.  And the way we do this is that it all happens automatically by integrating machine learning  tools so that if we use those machine learning tools, they will automatically upload all  the details of my research online.  Most importantly, it will link all results.  If I run a machine learning study on certain data sets, it will all be automatically linked  to all the other studies that have been run on the same data set.  That means that I can look at, I can share particular data sets and I can challenge everybody  in the world to build the best machine learning workflows on top of this and I can always  check which are the best techniques.  It also means I can do these large front searches like in Metacusar and share all this online  and they will be automatically organized online so I can just anytime go online and see what  people have been doing.  It could be that someone in the world as a robot scientist there making computations  or some algorithm that runs continuously machine learning experiments and I will always be  able to get the up to date results.  And very important that this happens mostly automated.  So for data scientists that use those tools, there will be no extra work.  They just have their username and password and they will be able to stream all the experiments  online.  They will be able to connect to other scientists, people they trust and share the results even  before they are published.  So I can start large cooperation with 20 people all over the world and they can see my experiments  the second they are finished.  You can see them online.  And what this, what you have to achieve with this is to tackle much harder problems.  So machine learning is becoming such a crucial tool in many sciences and many sciences are  tackling very hard problems.  And there is so much more we can do if we connect ourselves to this network.  There is a lot of latent expertise.  I may be working on a certain study but I may not know some method or I may not have resources  to run some method.  And by using OpenML, by connecting online, I can use this latent expertise.  Somebody else in the world may have just the right expertise to help me at that time.  So let me show you this works.  So first of all, because many people don't really believe that we can actually change  the way we do science.  And if you look back at history, you can see science has dramatically changed.  The way we do science has dramatically changed over time.  In the time of Galileo 400 years ago when he discovered Saturn rings, he didn't write  scientific paper.  For him, having the knowledge that Saturn has these rings is much worth, much more to  him if it's a secret than when he shares it.  When he shares it to him, it's basically become worthless.  He never shares it.  The only thing he did is write down his discovery, then jump all the letters into an engram and  he sends this engram to all his restaurant arrivals.  What is insured to this is if somebody else makes the same discovery, he can go back and  say, see, if you unscrambled these letters, you can see I was first so I should get the  credit for that convention.  This was how many people like Leigh Rondo and Avinci worked.  For them, having knowledge as a secret was worth much more.  This changed mostly through the work of the Royal Society.  The Royal Society was a group of researchers and then most of us think nobody's worth for  it.  They said, you cannot just make a claim and assume that we will believe you.  If you make a claim, you should come here and show the experiments and explain your methods  and then only if we see it actually happening, we will believe that the claim is correct.  They started journals to be able to disseminate both the findings and the methods.  That meant that all the other people in the world could reuse the same methods.  That meant that you didn't have to be incredibly relfal or you didn't need to be a genius to  do science.  You could actually look at what other people were doing and build on top of that.  They implemented this method in the best way of disseminating information at times which  was a printing press.  No, it was a giant advance for science because science exploded.  But 300 years later, you can ask yourself the question whether journals are still the  best way to do this.  Right now, we deal with very complex code.  We deal with huge data sets.  If we publish results in papers, they detach from all that code and data.  So you should work on systems where this is all integrated.  Also, in a paper, there's no room to put all the experimental details.  If you want to reproduce results, you have to make guesses.  Papers are not updatable.  You have to hope that other people will cite you and that the set-up is counted correctly.  There's publication bias, negative results are not published typically.  There's no way to really discuss a paper online.  Although recently, scientists are using Twitter to discuss scientific papers.  It's very interesting.  The best discussions of scientific papers happen on Twitter.  Open science is still, I think we're moving in the right direction, but still a question  mark whether we can change this journal industry to be more open access.  And you still do this while we have this internet and there's very great tools.  You can collect results to open online databases, to collaborative tools for code development.  You can build web service API to easily collect data from many different places.  We can use scalable algorithms to mine two incredibly large data sets.  We can go online and have discussions.  We can even include citizen scientists to help us if we have particular problems.  And many scientists have already been thinking along those lines.  One of them is, for instance, Tim Gowers.  He's a field medallist, right?  One of the great mathematicians of this age.  And he had a hard problem in mathematics.  He decided one day he was stuck on it and instead of just putting it in his drawer, he said,  okay, I'm going to just put all my partial information online.  And everybody in the world can look at my train of thought and can add to this and maybe  we can collaboratively solve this hard problem.  Within a month, many, many people have been having active discussions and within 40 days  have actually solved the problem.  They actually solved an even harder generalization of the original problem.  And what this does is it actually combines the many minds towards solving a particular  problem.  And if you go, if you have to find out, I need those and those and those people to have  the right expertise, that's very hard to do.  But if you work openly, if you say, okay, I'm solving this problem, if you want to help,  you can help me, you kind of attract the right expertise from people and by working together  you can actually solve much harder problems together.  And the way they do this, they make jump publications out of the proofs that they found based on  who contributed what to the findings.  It's also apparent in the SDSS, the Sloan Digital Sky Service.  This is a robotic telescope that slowly scans the universe.  And very interesting here is that after one year, they put all the data online.  That means everybody in the world can just say, okay, I want to look at that part of  the sky, those wavelengths, and you can get imagery from that part of the sky.  This is a huge resource, this is basically one of the best resources in astrophysics.  Many people have been using this to answer questions.  And the idea here is that you broadcast data hoping that other people will have interesting  things to discover in the data.  Because when you collect data, you don't always know exactly what kind of questions  you can answer.  But somebody else may think, hey, if I had that data, I could answer these kind of questions  much quicker.  This is what the SDSS basically does.  And this works incredibly well.  There's thousands of papers on new discoveries, reducing this data.  For instance, there were two astronomers thinking about what is possible for two black holes  to overdure on each other.  And instead of running very complicated and expensive simulations, they actually downloaded  data, they used machine algorithms, and actually discovered this happening in space.  Most interesting, while there are only about 10,000 astronomers, they have only one million  people looking at the data.  That means that 99% of the people looking at the data are not astronomers.  They are school teachers and people, amateur astronomers.  People are just interested in looking through space.  And sometimes they make new discoveries.  One of the discoveries happens through Galaxy Zoo.  So one of the problems that they were having is that they have about a million galaxies  and to analyze them, they need to classify them.  This is a left-turning spiral galaxy or right-turner or it's like a beam or a ball galaxy.  And instead of hiring PhD students to manually annotate all these galaxies, they made a website  and say, look, everybody can go online, you get a picture of a galaxy and you can classify  it.  It was usually successful.  I think at some point they had 70,000 classifications an hour.  All the people in the world, like looking through these galaxies, it blew their servers.  But this really shows that if there's interesting information, people will be interested in  looking at this and especially at the rates that we collect scientific data these days,  we collect data faster than we can actually look at it.  So you as a citizen have actually a chance of seeing a galaxy for the first time.  You can be the first human being to look at this galaxy.  And it's the same happening in biology.  Maybe you're the first human to look at a certain brain scan or whatever that's being  shared through open science.  So one of the things we discovered is you have these little green blobs sometimes.  And most professional astronomers thought this was just noise.  But then it was once a Dutch school teacher who said, I don't think this is noise because  I've seen this many, many times.  She started a blog.  People started looking for this.  They found many new examples of this.  And this turned out to be a new kind of galaxy.  And they've discovered about three new types of heavenly phenomena in this way.  And they do the same thing.  You can go online.  You can hunt for planets around missing stars.  You can look at plankton to discover new types of plankton.  It's really amazing.  So how come that these projects, by working more openly, make new discoveries?  What is there different from doing the normal way of doing research?  Well, this can be best described as this design to interpret it.  It means that by broadcasting data and broadcasting questions, you can make spontaneous discoveries.  By just the right person looking at this, having an idea, or having just the right expertise  to interpret the data.  And by doing this, you make discoveries otherwise are missed.  If you are a scientist, you have a certain mindset, you may overlook things easily.  But if you share the data openly, more people will be able to find those, to interpret it  correctly and make new discoveries.  Also, it's hard for one scientist, easy for another.  I can be working on a study for, say, I'm a bioinformatician.  I'm using this shielding algorithm and getting better results.  I don't know why.  And instead of having, if there's somebody else who, maybe the guy who designed the algorithm,  or maybe somebody else who knows a lot about it, may say, oh, I know it's going wrong.  You should use, first of these parameters, or you should reformat the data in some way.  If you work online this way, you may be able to find the right person to actually help  you in your research.  Also, what's surprising for one scientist, say you have a weird result, and you probably  cause it away, for another scientist, he may be able to explain exactly what's happening,  or he may have a new idea that sparks new discoveries.  And this network effect means that you can do many more studies that were previously  impossible.  If you want to give just so many things you want to do, and you have so little time, by  working more openly, you can say, OK, I have this interesting data, this interesting result.  I don't have the time to look at it, but I'll share it anyway.  And maybe somebody else may be able to do something interesting with it.  And maybe I have a problem, and I don't want to wait a year to get a grant to analyze this,  but I can maybe work online with other people and do this much more quickly.  By working together, these studies become much more cheaper, much more faster.  Now to do this, you need the right tools, you need the right online tools to communicate,  to share results, and to share at the right level of detail with each other.  So it means you have to remove the friction that's currently very common in scientific  research.  We do experiments that we email them to somebody else and have to reformat it and look at it.  And it's all very high friction environment, and this kind of inhibits this rapid interaction.  You don't want to spend an half a day like putting two days together.  If it's already there online, you will be much more inclined to look at it.  So it means that you have to organize the data into one coherent whole.  If you have data from one researcher, another, another, another, it should be easy to take  all that information together.  I'm not going to spend a day looking at what each of them did and then making them together.  If you have an online system that does it automatically, I can easily use the results  of thousands of researchers and combine them in new studies.  It's also important to make microcontributions.  It should be possible to upload a single data set, to do a single experiment, or to make  a single comment in a scientific discussion.  Because many people won't spend a day looking at something and then making a comment.  They want to do it more quickly.  It should have easy organized communication, so we can actually communicate in real time  with each other.  There should be a clear honor system that if you work with somebody else, you give them  credit and the system should enforce that.  The people don't just run away with your data.  If they reuse it, you are being credited for this.  You can do this automatically.  If you use your data, you can automatically track this.  It should also protect preliminary work.  I want to work with, say, I'm working on a problem like finding new drugs.  I don't want the whole world to see this.  I need to be able to work together with a group of people that I really trust.  Those people can see my latest results.  Only after the papers published, or the patent is published, we made this data openly and  from there on everybody can use it.  The important thing is that making a public is just a push of a button.  It doesn't mean I have to spend two weeks doing this stuff because I'm not paid.  I'm not going to get extra credit for making this data nice to be used by other people.  If it's from the start, if it's nicely organized, just a click of a button, everybody can reuse  and build on this data.  It also means more people will reuse your data and it also means that more people will  cite you and this gives you, again, more visibility and more credit.  We should find new ways to measure impact.  This is why we developed OpenML.  The way this works is anybody can upload data.  Say your data scientist is working on a problem, say, for instance, there's data from the  Metacuser project.  Just upload it online.  You cannot see it because it's still private.  It's on there.  You can say, this is my data.  This is what I'm trying to do with it.  This is what I'm trying to discover.  Everybody in my collaboration or everybody in the world can then help me do this.  So once you upload the data set to OpenML, it will first analyze the data.  It will extract meta features from the data.  If it's fingerprints, it may look how many features, how many columns are there, how  do landmarks perform on it.  If it's structured data, you may have some information about the structure of the data.  If you upload data to OpenML, it will try to extract information for you.  So you can always link the results to the properties of the data.  It also makes a structured index where you can easily search for the data and it makes  machine-readable tasks.  So it says, okay, this is data.  This is what I want to do with it.  And you make a little XML file so that software tools that can use this can automatically  interpret this.  So then scientists, they can design workflows, get solved a task, and then they can upload  the predictions.  And the way this works, I typically have my software tool.  I say, okay, run on this task.  It will download the task.  It will download the data sets.  It will perform the data analysis.  And yet it will upload the results back to the server.  And then I can go online and see what's been happening.  So if you upload the results, OpenML will store the full details, the full workflow,  the prem settings.  I know exactly what I'm doing and everybody else can see what I have been doing.  So all results are easily reproducible.  It analyzes the results.  In some cases, like classification, you want a score, like how good is this classification  and OpenML will compute these scores.  And it will organize results online and visualize them.  And when that's done, scientists can still go online and then reorganize the results  and then they can discuss the results with each other, clean it, link it to papers, and  such on.  So the OpenML works.  This is how it looks.  So you have a number of data sets.  So let me just look for data sets.  Say the cover type data set.  You can type in cancer or malaria and you can see what the assets exist about the data.  So, say upload the data sets.  So then first of all, there's the description of the data sets and it's like a wiki.  So if you click the edit button, you can automatically add information.  So if you know more information about data sets, everybody can edit this description  and make it more complete.  Then the server computes properties.  So you see how many instances, how many features there are.  But it also has a library of different ways to annotate these data sets.  So you can see it calculates a whole list of features about the data sets so you can  relate the results of your machine learning performance to this property of the data set.  So in this way, this is important for instance, if you want to know which kinds of algorithms  work well and which kind of data, then you can do this meta-learning studies.  So you can pair these features to the performance of your algorithm and then you can build decision  trees or whatever to learn relationship between the features of the data, the props of the  data and the different machine learning algorithms and how well they do.  And to also do some statistics about the data sets so you can see the class distribution.  You can look at all the features and you can see how values are distributed there.  So maybe you find something funny in there like a value that's not normalized or a feature  that basically uses because it always has the same value and so on.  And then below you have the tasks.  So once you have a data set, people can design tasks on top of that.  So one task would be classification.  It means here's a data set, here's a target feature.  Now build me a model that predicts the target feature.  So if I click at this time, so here I see a classification, I have data streams, I have  learning curves.  So let's take classification.  Now it's collecting all the experiments that people run on this data set for classification.  So it gives description of what the classification is supposed to do.  It gives you the inputs like the cover type data sets, the constellation you need to use,  the target feature and so on.  And these are the output sets that are expected from you, like the evaluations, the models  and the predictions.  And then you see, okay, so 126 runs have been computed.  You can score all these runs by different evaluation metrics, depends on what you're  interested in.  Here you see an overview of all the different machinery methods that have been run on this  data.  You can see how well they performed.  And each one of the dots is run, is a model that people build.  It's a bit hard on iPad, but if you click on this, you get all the details of the run.  Okay, right, so, okay, you can see who has run the experiment when, what's the algorithm  that they used, which is the parameters that were used.  You can download the full predictions, instance on instance level predictions.  You can look at the model, you can actually download a serializable binary of the model.  So you can actually reuse the model to make new predictions.  And then you see the evaluations that the server computed on the data.  So area on the rock curve is 95% and so on.  It took 2600 seconds to build this model and such and such and such.  So what OpenML did here is somebody has given a data set, asked the classification with  it and here you see all the results by all the people who have run classification on  this data set.  You can do the same for stream mining algorithms.  So here I can look at data stream algorithms, here I get a different task, there's a different  set of algorithms that are being used to solve this task and so on.  So you can define any kind of task like clustering, you can have a special meta-qsr task to look  at only those results and so on.  Okay, so you can search anything.  Okay, so there's an, okay, I'll just switch stuff.  You also have the details of the workflows, right?  All the experiments that are visible are also repeatable, it means you can download the  data, you can also download the code that's being used, right?  So say I'm interested in running forests.  So here I can see how running forests performed, again I can choose a performance metric.  I can see how it performed on different data sets, that's quite a few, and I can also see  the variance.  So every point here is a model and the variance is caused by different parameter settings  of the random forest and there's also a little help to visualize the effect of a parameter  and here you see, so the red means random forest, a large number of trees, blue means  random forest is smaller, you can see what the effect is of the number of trees on a  performance over different data sets.  You can use four parameters settings.  Okay, so this is the data that's been organized.  The most crucial thing is that this happens automatically, right?  So people don't want to upload manually upload data, it should all happen automatically,  right?  And so OpenML has a web API, it can be linked to data resources like QAML, like QCi, like  ML data where it can extract the data sets from, it will then make them easily available  and easily searchable so that you can do large scale experiments.  It also links to code repositories like tools like Weka, like also like code on GitHub,  if you have code on GitHub you can also just link it to the GitHub page.  It has a Java API to connect it to Java based machinery tools like Weka, Moa, Webminer,  NIME.  It also has an R API to link it to R packages like MLR and data mining with R or the next  package that Liz will invent.  So you can go online, you can look at the web API, explain quite well what you need to  do to talk to the server.  There's of course a Java API, in that case you just have, you create a Java object client  and then you use a client to talk to the server with a certain set of methods, same with R.  In a few lines of code you can actually do an experiment, download, run an algorithm,  upload the results again.  There's also what's being used in the QSAR project.  It's also directly integrated into graphical tools like Weka.  So if you use Weka, the next version of Weka will actually have it integrated.  So you just select the OpenML experimenter and then for the result destination you select  OpenML, then you can give a number of tasks, you get a number of algorithms and then Weka  will run all the algorithms and all the tasks, automatically upload all results to the server.  Same with Mova, you can select tasks, you can select algorithms and automatically all  results will be uploaded.  Online, say Webminer, you can have an operator, it's called Node, that actually downloads  the data, then you can build your workflow at the end, you have a separate operator that  again uploads all results online.  So what this means is that all the people that do data science research can use Weka,  Webminer, R and with a little extra effort to talk to the server, they can automatically  share all their results online.  Let me say everybody else can reuse their results.  You can set up large collaborations and get second by second updates on what everybody  is doing and all the information is shared online.  You can build on top of each other's work.  What are the benefits for scientists?  Why would they do that?  First of all, you gain a tremendous amount of time.  There's always a tail of scientists telling, well, 10 years ago when I was a young researcher,  we had to do this and this and this manually.  Hopefully in the future, all of these things will be automatic.  It means finding data and code should be just like a search query and you can see all the  interesting data sets, all the interesting code.  You can run that against that and have all the experiments automated online.  That means you can run large scale studies.  I can do huge parameter sweeps and have all results organized online and just doing queries  and I can build models.  I can build better models on top of that without having to do all the stuff manually.  We used to do this stuff in Excel.  Nowadays we can do it much, much better.  We can immediately compare to the state of the art.  I don't have to email other authors, so please give me your code so I can compare against  it, now I can just compare against everybody who shared the results.  I can immediately compare against them without actually running their code.  I can just reuse their results and if I'm skeptical, I can always rerun the code because  the code is available and the data is available.  So it gives me a lot of time and I can do many more studies that were impossible before.  It leads to more productivity because I can directly work, I can build directly in prior  work.  I can look at an interesting paper and I think, okay, I'm interested in this.  I want to do more experiments with this.  Let's download the data and let's do interesting stuff with it.  This wasn't possible before.  You needed weeks, maybe months to reproduce the same experiments.  You can work in online teams.  If you need expertise, you can invite other people to look at this data and you don't  have to email them data.  They can just go online.  You just say, okay, add this person to the collaboration and they can look at all your  data, visualize it whatever they want.  You can quite source problems.  So if you have a problem, for instance, you have this data that you need to process in  some way.  You don't have the right expertise.  You can always call people to help you.  You can quite source all these problems.  Overall, it gives you more visibility, right?  Because especially outside the sciences, because if you talk to people with industry or the  general public, they will not read scientific papers.  Most of them won't read scientific papers.  But here you can just go online and look at all the algorithms and you can see how well  they performed.  So it's much easier for people, especially for students to see what exists, what's the  state of the art, who are the best people in this area.  And we can actually rank people with leaderboards.  It also gives you more reputation because by sharing your data, other people can more  easily reuse it, which leads to more certations.  OpenMell can actually tell you if you reuse data how they should cite you.  You can ask somebody, we use my data, this is how they should cite me.  And OpenMell can also automatically track if they reuse results in a different study.  OpenMell can see, okay, this study contains experiments from these 100 people maybe.  Right?  You can track that.  And also by sharing results, you build trust, right?  If you publish a paper and the reviewer can actually see all the data and all the algorithms  this gives the reviewer all the more reason to accept the paper because clearly you have  done all your work correctly.  For visibility, OpenMell is used all over the world.  So this is the statistics for August.  So the big group, this is just, no.  This is me and Jan working on it.  But I mean, people all over the world are at least checking what information is in OpenMell  and what data they can reuse.  Who said the Far Far East?  Yeah.  Even on Hawaii.  This is interesting.  These little islands.  I don't think you was here actually.  Yeah.  Okay.  You have to work on Greenland though.  Okay.  There's also huge benefits for students.  If you're a PhD student, you can easily get an overview of the state of the art.  If you want to get noticed, you can participate in these large scientific problems and show  what you're worth.  You can get in touch with the big scientific minds of your age.  You can ask questions and students can help each other to answer questions like, like  happens on Quora, but here's like specific for machine learning.  Okay.  Things that will appear soon, I hope at the end of this week, will be OpenMell studies  so that you can just say, okay, I want to have all results on this, on this data sets  and this code.  I build a study on this.  I write a description of it.  And if I publish a study, I can just put a link in the paper that links back to the  special page where all my experiments are and everybody can look at that.  I also want to reorganize data by tagging.  So say I want to do a support factory machine study.  I can tag the different algorithms with support factory machine and I can then build a study  out of that.  I also want to work in teams.  So it's already possible to set up a private team of people to work with, but we want to  have more support for that so you can easily add more people to the collaboration.  Right?  We also want to work on reputation building.  So if you share a data set or you share an algorithm, you can see how many people are  looking at it, how many people are reusing it and such.  And what we definitely want to go to in the next, not this week, but in the next couple  of years maybe, is to go towards kind of a robot scientist for machine learning.  So in a situation where you upload a data set and there will be clever algorithms that  say, okay, these are the most interesting machine learning methods to try and it will  use algorithm configuration techniques to fine tune those methods to work best on the  data.  And you can do this because if you share all this information about different workflows,  different machine approaches, different program settings, you can build metamodels that actually  predict based on the features of the data which methods may be more useful.  And that way, I hope it will be much easier for scientists to get help from the system.  You can upload a data set, you can ask the system, give me your best guess what's the  best method you can do against this.  And you can actually do like a challenge, right?  You can actually put humans against the machine, where the machine says, okay, I want to use  this workflow and this is the best result I get and you must try to do better than the  machine.  It will be very interesting.  Okay, that's about it.  I would say you are already part of the club, but feel free to join.  This isn't very old sort of pictures.  We should take new pictures.  And yeah, I look forward to a great week of working with you and if you have questions,  just shoot.  Thanks.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.
OpenML2015-OpenML-Vanschoren_0_227.png OpenML2015-OpenML-Vanschoren_228_4163.png OpenML2015-OpenML-Vanschoren_4164_7811.png OpenML2015-OpenML-Vanschoren_7812_8555.png OpenML2015-OpenML-Vanschoren_8556_10415.png OpenML2015-OpenML-Vanschoren_10416_14939.png OpenML2015-OpenML-Vanschoren_14940_17267.png OpenML2015-OpenML-Vanschoren_17268_17771.png OpenML2015-OpenML-Vanschoren_17772_24419.png OpenML2015-OpenML-Vanschoren_24420_26087.png OpenML2015-OpenML-Vanschoren_26088_39719.png OpenML2015-OpenML-Vanschoren_39720_42719.png OpenML2015-OpenML-Vanschoren_42720_43295.png OpenML2015-OpenML-Vanschoren_43296_44891.png OpenML2015-OpenML-Vanschoren_44892_45023.png OpenML2015-OpenML-Vanschoren_45024_45455.png OpenML2015-OpenML-Vanschoren_45456_45599.png OpenML2015-OpenML-Vanschoren_45600_45671.png OpenML2015-OpenML-Vanschoren_45672_46343.png OpenML2015-OpenML-Vanschoren_46344_46715.png OpenML2015-OpenML-Vanschoren_46716_47435.png OpenML2015-OpenML-Vanschoren_47436_48107.png OpenML2015-OpenML-Vanschoren_48108_49043.png OpenML2015-OpenML-Vanschoren_49044_49151.png OpenML2015-OpenML-Vanschoren_49152_50711.png OpenML2015-OpenML-Vanschoren_50712_50831.png OpenML2015-OpenML-Vanschoren_50832_50939.png OpenML2015-OpenML-Vanschoren_50940_51455.png OpenML2015-OpenML-Vanschoren_51456_52487.png OpenML2015-OpenML-Vanschoren_52488_52583.png OpenML2015-OpenML-Vanschoren_52584_52631.png OpenML2015-OpenML-Vanschoren_52632_52955.png OpenML2015-OpenML-Vanschoren_52956_53813.png