Project: Egkatastasis
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Georgiadis, Panos
Testing package installation using containers In Tumbleweed with have roughly ~25.000 packages for 64-bit architecture. Do you know how many of those are actually install-able? From those who are not, do you know the reason behind? Do you know how many of those will become install-able if boo#123456 gets fixed? And from those which can actually be installed, do you know if there are any glitches at the post-installation scripts? -Sure, we have openQA, but still, it tests only the packages inside the DVD and not the entire ecosystem. - Sure, we have the OBS. So, everything that gets build should also be install-able. No? -Sure, we have libsolv techniques that can answer this. But have you tested if the results reflect the real world? There's only way to do verify what's really happening: one system per package. Yes, that is extreme, you would probably need 25.000 virtual machines. But ... hold on... what about using containers? Well, I have an idea! I have developed a project for fun, and I would be delighted to share it with you. Egkatastasis (you can call it *egg*) is an open source system for testing openSUSE container images providing basic mechanisms for installation, log analysis, and metrics visualization of every package contained into the official repositories. Egkatastasis tests production container workloads at scale using Docker and systemd-nspawn, combined with the best-of-breed ideas and practices from the community using Filebeat and Elastic Stack.
 Hello, everybody. Welcome again. My name is Panos Uriadis. I'm QA maintenance engineer for SUSE. Today I'm going to talk to you about a personal project of mine that I did during Hack Week that I received a lot of good feedback and interest. I'm here setting it with you. The project is called the catastrophes, which is a Greek name. Most people couldn't even pronounce it. The reason I picked the Greek name is because, why not? I mean, since I'm in the containers world and testing, everything looks to be Greek there. Even Kubernetes is a Greek word. So I was like, I will use a Greek word. So what the catastrophes means for the people who are interested in that, because Marcus asked me before. Well, if we split the word, it's actually N, which is in, Kata, which is two, and Istimid, which is stallum, which is Latin, means medium. So it moves something into a location or medium, which is install. If you just look at that, and then here is the installation that everybody knows. So a catastrophes means installation. So my project is about installing something in that case. What it is? What it is? Yeah, I just came up with that. It's an open source project. You can find it in GitHub. And what it does, it's supposed to test packages inside the open source containers that we see. Okay. And how we do that? I use Docker and SystemDnSpawn. So I use both a process container and a system container. And I use then the Elk stack and the file bit. The whole project is in GitHub. And in case you like this, feel free to contribute to that and let's see if there is more interest about it. So how it started? About one year ago or even longer, it was HACWIC. And I was checking containers. I was like, what is Docker? What it is? I was learning stuff. So basically, I said, okay, I am an open source guy, so let's use the open source containers. So I go to GitHub, I pull the Docker image we have for Tumblr. And I see that we have orphan packages there. Like if you do Zipperserts in the package, this package was coming from nowhere. And then I filed a bug. We talked with Martin, we talked with Flavio, and we make it fixed. But it was a bug. Next, we had wrong repositories. After that fix, three months later, I downloaded again the new image for open source container trying to do Zip and refresh and Zip and refresh fails because the URL in the repository was wrong. And I was like, what is going on? We have packages coming from nowhere. We have broken repositories. How people and why I am the only one that finds those stuff? Like there was no bug report. So after those, I started looking how I can test it. So a lot of people how I can test containers in OpenQA. And the feedback I got is that basically you have to install OpenQA first, and then you have to spawn a virtual machine inside this virtual machine to run in a container. Which looks like too much. Like it's way more things to be done in that case. And it's really, really slow compared to the typical testing of containers. For those who are interested, since we are in the university here, there is a Google summer of code project for that. That people and students are interested to help us and contribute to OpenQA by investigating a docker or a container backend in that case. So feel free to check this. And so I talked with the guys that we do testing. I saw also this and then I realized that, you know, we actually push open-source images one year ago with no testing them. So I was like, let's try to see how I can test them. That was the motivation behind the whole thing. So the project tries to answer how many packages might be problematic without my knowledge. I mean, I was just lucky when I did the Zipper search on that specific package, which was Kerberos 5 mini in that case. But can this be that there are also other packages that are broken that I'm not aware of? And if so, how do we know unless I test them all in that case? How many packages are supported in our open-source container images? Are the same packages that we support in Tumbleweed, all of them supported also in open-source containers? I mean, if I open a bug, would be a legitimate bug or they would say, no, it's containers. We don't really support it there. And how big undertaking is going to be this, to test everything? How much time I'm going to take for that? Since it's a community-driven project, it's just for fun, learning, experimenting. So I would like not to spend that much of time. How much resources I'm going to need for that? How difficult is going to be? And how accurate the results are going to be? That was some first questions that I had. Okay? So expectations. The expectations I had is like, this thing has to be fast. What fast means? Fast, definition of fast for me, is in that case that I'm going to test the whole ecosystem of open-source packages. This means roughly 50,000 packages. I need to test individually 50,000 packages. So in that case, I would say a week would be the maximum for me. I wouldn't wait more than a week to see a result, five days or something like that. Trust. Trust means if I find something that is broken in the container, I expect that the same behavior happens in your laptop. If this is not true, my whole project is... No, it's working. Okay, perfect. Okay. Yes, I was saying that the whole project makes sense only if what I find is real and it's a bug officially and we have to fix it. Otherwise, if I see different behavior in a container that I've done in a laptop, on a desktop, it doesn't make sense to do it. And it has to be simple. Simple meaning anybody that wants to contribute, anybody who's not familiar with containers but can jump into it really fast without much knowledge. Requirements for the test environment in that case. I had to build an environment in order to test that. So infrastructure as a code. I needed something that no matter if I'm sitting in my laptop in my workstation at work, if I visit my friend that has another distribution, I would like not to spend time installing and configuring from scratch and messing up his system. So everything, the whole project is containerized. You just get cloned, run one command, you don't infect your system with any of the things I'm doing. It should be low overhead, meaning like it should be able to scale. If I run it in my laptop, which is a dual core with hyperthreading and 4 gigs of RAM, I should be able to run the whole project in my laptop. And of course, if I move the project into a different infrastructure which is bigger, 20 cores, more RAM, the project should be able to take advantage of this and be faster. Right? And then I needed isolation for exactly what I told you before. Like, I don't want to mess my host system when I'm doing testing in that case. So for isolation, what we have, we have virtual machines that they offer great isolation. But the tooling for making this automatically was a little bit not the easiest, let's say. As I said before, I would like to have things really easy for that project, like one command things. And also with the virtual machines, I realized that the project is not going to be lightweight. Because if I need to have one package per virtual machine, this means I need to spawn 50,000 virtual machines. This means I will end up in 2030 in that case. T-route, yeah, quite the opposite of what the virtual machine is, terrible isolation, terrible. So forget this. And I ended up with container with something in between. They offer, okay, isolation, the container can see, the kernel of your system can access the hardware resources, but at the same time is not going to mess my host or packages in that case, only touch the file system of the container. So I ended up with containers also because it was time for me that I was experimenting. The methodology now here. Test, student, influence each other. What it means. Back in the day, I was a journalist writing Linux articles and tutorials and guides. And one thing that I really learned is that if you write a guide for somebody to follow, you have to have a clean system always. If you are going to provide information on write documentation for somebody, you should make sure that you download the ISO, you boot a virtual machine, you do what you have to do, and this thing should be reproducible. So in that case, let's say that we have a test that I have to test packets A. So we spawn a virtual machine in that case, I install packets A. Then if I continue in this system and install packets B, it might be that packets B has a conflict with packets A, but this is not clean anymore because packets A is installed. It should be a virtual machine, packets A. Another virtual machine, packets B. Something like that. So one test, the test for testing packets A should not interfere with test of packets B. That is the scenario. Test should be ephemeral, meaning as soon as they run, they get automatically deleted. They don't take space in my machine, they don't use my resources anymore. I don't want to manually take care of removing containers, their images and other logs of Docker in that case. Test should not affect the host as they are running, what we said before. So this is a screenshot. Yeah, not the best one. I'm using Timux here, but I can give you a demo later on when I saw you better. What I'm doing here, basically, you see every line you see here, every of these green lines that has success, which means open suzerox in that case, all the packages in that screen is installable, is one container. And this in that project, here you see, for example, how much time remains until the test is complete. Here you see file bit harvesting for logs and sending them to Elkstack. And here I have wrote a really, really high-key parser, since I'm not aware of any parser for zipper logs. Maybe if you are, please let me know. So I just look for errors and I parse for specific strings in the logs to see if they are happening. So the last time I ran that was in January. And we find, I found out that basically 98% of our 64-bit packages we have in open suzer is super healthy and only the other percent is not installable. The good thing with that project is that even if a package can be installed, sometimes we have messages happening in the post installation scripts. And you can aggregate those and see exactly how many packages are the same messages. And basically, when we release an open suzer snapshot, if we run this after that, we maybe can say, okay, if we fix this specific bug, 10 or 50 packages is going to be also fixed. So let me continue. Use cases. Visualization and metrics for the installation of our packages. You can either use the CLI and see the results on the command line, but you can also use elastic search and Kibana and go to your release manager and give him graphs and bars and I'm sure he will like it. Then I was like, what if open suzer had serverless infrastructure that a user before go to Facebook, Reddit or open to the forums and start complaining that package A is not working in his machine because he has installed a bunch of custom personal repositories from other people, he can go something like broken forever or it's just me in that case. So he can just, for the people that were in the previous presentation that I did, he can just send a request with a package and he can see if this package is installable or not and if it is in his system, it's not, then there's something wrong in his system, like he shouldn't file a bug for that in that case. Integration in build service, I'm not sure about that, I'm not very familiar in build service, actually I'm not at all familiar with that. I assume that there might be a use case for that as soon as we build the package, we can just run this outside of build service in another infrastructure in order not to cause any overhead in the build service in that case and have the output in a database. So this can be run also during building in that case. And this is a good entry point for newbies like me in that case, that they want to be maintainers of some packages. When I was volunteering for the Ubuntu bug squad, I remember that they had a panel that you can go as a noob basically and say, you know, I want to contribute to this project, I don't want to do translations, I want to write some code, they want to be a maintainer, but I have no idea from where to start, I need some easy tasks. And with this project, since you aggregate all the problems that might happen in the packages, you can classify them as easy things to do, like trivial things to do, and somebody you can give it, you know, take this and fix it. So there might be a collection of easy to solve problems for the people who are passionate in Ubuntu as maintainers. And then we probably can have like a comparison. We can have a live comparison of the status and the quality of the installation of packages in SUSE, Fedora, Ubuntu, Debian, you name it. You can convert them and we can also check live what version of packages they have. For example, I see people in forums say, we have the latest GNOME, no, no, we have it, but yeah, but we have this and that. So with this one, if you have all the logs there, you can actually take this information there, you can have a graph and everything. So I will talk a little bit about the future plans that I have with that and then I will give you a demo. So I will try to put this in Kubernetes to have it in a fast way, like you have a function, you say test this package, this package gets tested and I get the result if it is installable or not along with the log of Zipper in that case. Really, really fast. And then I would like to experiment with that, but now that Markus told me that he doesn't like this, you have to reconsider. Maybe with Cata containers, Cata containers and Google device or would be the thing that I would like to play just for experimentation. This is a fun project. Then I will try to learn a little bit about packaging and build service and see if it makes sense to put it there and how it works. And again, what we said before, entry point for Nubis and comparison among other distributions. So I am basically planning to have a website for that thing that you guys visit and you see results. So that's the presentation. Let me give you a demo with that. So what you have to do is clone the repository. The repository, I have already cloned this in that case. So let's go here. Let me open my browser. So here, if you go to GitHub, I basically have a tutorial, MD for the people who would like to try that. And also in the very front page, I have a logo though. So it's professional. You can just go into this folder, Docker in that case. So we go to, let me zoom. And here, there is a bunch of scripts. Let me see how much time we have. Okay. Another 10 minutes. So you can do test it. It should be simple. Like that. It will not infect your packages. And you put the name of the package that you would like to test. So what's happening in that case? In that case, there is an open-suzet with container downloading in my machine, running up, running the zipper, install Vim, as you know, with non-interactive accepting licenses and everything because I can't interact with that. As soon as this is finished, it will kill completely the container and remove it from my machine. But it will take the log from zipper and store it locally. As soon as this log is present in my system, file bit, here is going to collect it and send it to Logstas, which is a storage for enterprise logging. Logstas will do some purely idiotic filtering that I'm doing because I don't know any zipper or parser for that. So I'm just taking the whole thing. And it sends this to Elasticsearch. And I have Kibana, which is the web UI for Elasticsearch in case you would like to have some graphs and stuff like that. Perfect. So we see here that Vim is installable right now. That's a live test in the Tumbleweed. So if I open, for example, that's interesting, opening Vim. Yep. You see this is the complete zipper log. So in our open-source containers, that's what will happen. Six new packages will be installed. And here is the log of the whole thing. This now goes to Kibana, as I said. So... Yes. Here you can see that file bit took the file, send it to Kibana. And I have here the whole log in case I would like to have a central database or have a website, a web server that feeds the content from that database. Okay. And I think that I have something here that you might like. Yesterday, I ran this in my workstation at work. And I can give you some results for the packages that they are no-arts architecture. Unfortunately, the 64-bit packages are about 19,000. And this takes about one day and a half. And since I started it yesterday, it's not finished yet. The only thing that got finished is there are no-arts packages that they are roughly 15,000, I think. So I'm going to take this file and put it into my project in that case. And let me unpack it. So now we will see file bit getting crazy reading 15,000 logs. This means like since yesterday, I've managed to run 50,000 individual containers with one package per container. So we know exactly what's going on in our open-suzet tumbling with container images in that case. Let me, since this is sending the stuff to the database, let's use the parser to see some stuff here in the terminal. So you see here the timeout. I was a little bit too happy about that. And I was like, okay, I have six core machine at work. Maybe I can run 200 containers at the same time. Yes, I can. But what happened is that my internet connection is not that fast. So basically, I run into the problem that I have a hard time out that I don't want to wait forever for a package. There might be packages, if we check those packages because the open-suzet container image is pretty, pretty small, there might be that if you try to install one, it will try to install three gigabytes of packages. Theo loves, I know he loves. So what happened is, yeah, internet connection problem. So just some things here. We see that some packages here could not be able to install. So if I do like grep for that, we can see those packages. Python Django, network manager, novel VPN, this looks like a very old one. Copana web hub, and stuff like that, and what is also interesting is from what are installed in that case, I have like 1,500 wrong permissions here, something like this. So if I search for that, I get that, like setting, blah, blah, blah, trusting wrong permissions. I'm not sure if this is a bug or not, but this is something that I get. This is not a bug. Then why it says wrong permissions? Why there is the word wrong? I mean, yes? Because the permissions file is a configuration. All right. So these are not bugs that's good to know, but at least what I'm trying to say here is that the parser checks those. In case you need them, to check them, the parser checks those. And all those things are now in the database. Hopefully, all things are now here. Let me refresh that. Okay. So, yes, everything is here. All the packages. Yeah. And you can go to visualize, create a visualization, create a pie chart, and here you can have your metrics like these are all the packages that we have, these are the packages that fail with that error, these are the packages that fail with the other error, and so on and so forth. So you can say, okay, classify them and give them to your user to fix them. So that's the project so far. It was really fun for me and I received good feedback, so that's why I'm here. Maybe it's interesting for you. You have some ideas to tell me how this can be improved. Any questions? Yeah, let me pass you the microphone over. This can open like this. Okay. Yes, I didn't dare to press the red button. So, whenever you submit a package to open to the factory, there's an install check, but of course, only for that single package and of course, it could be that it breaks another package that's already there. So maybe these are the errors that you can find with your approach. But on the other end, when we have like a daily tumbleweed snapshot and you say that this takes like, you know, about a day, we cannot run this test for every tumbleweed snapshot. Do you have a suggestion when to do this? I don't know. Maybe we can write it for Leap. I have no idea if it makes sense. Actually, we spoke about that already in the past. You could only test the packages which changed and you could talk to a WebMQ which is fed by the build service, which packages are changed and you only need to test those. One thing that your regular test would find is packages which are fair to build and are now broken because their dependency has changed. So you would still need regular tests like once a week or once every two weeks. But if you only want to test in the tumbleweed time frames and test the packages where you get successful build events or publish events. All right. Okay. We have one minute. I would just like to say that this thing works with system dn spawn container also. So that was another experiment. You can find details how to do it. Any other questions in the room? Yes. No. No. Okay. So no questions. Thank you very much for being here.
 Hello, everybody. Welcome again. My name is Panos Uriadis. I'm QA maintenance engineer  for SUSE. Today I'm going to talk to you about a personal project of mine that I did during  Hack Week that I received a lot of good feedback and interest. I'm here setting it with you.  The project is called the catastrophes, which is a Greek name. Most people couldn't even  pronounce it. The reason I picked the Greek name is because, why not? I mean, since I'm  in the containers world and testing, everything looks to be Greek there. Even Kubernetes is  a Greek word. So I was like, I will use a Greek word. So what the catastrophes means  for the people who are interested in that, because Marcus asked me before. Well, if we  split the word, it's actually N, which is in, Kata, which is two, and Istimid, which is  stallum, which is Latin, means medium. So it moves something into a location or medium,  which is install. If you just look at that, and then here is the installation that everybody  knows. So a catastrophes means installation. So my project is about installing something  in that case. What it is? What it is? Yeah, I just came up with that. It's an open source  project. You can find it in GitHub. And what it does, it's supposed to test packages inside  the open source containers that we see. Okay. And how we do that? I use Docker and SystemDnSpawn.  So I use both a process container and a system container. And I use then the Elk stack and  the file bit. The whole project is in GitHub. And in case you like this, feel free to contribute  to that and let's see if there is more interest about it. So how it started? About one year  ago or even longer, it was HACWIC. And I was checking containers. I was like, what is Docker?  What it is? I was learning stuff. So basically, I said, okay, I am an open source guy, so  let's use the open source containers. So I go to GitHub, I pull the Docker image we have  for Tumblr. And I see that we have orphan packages there. Like if you do Zipperserts in the  package, this package was coming from nowhere. And then I filed a bug. We talked with Martin,  we talked with Flavio, and we make it fixed. But it was a bug. Next, we had wrong repositories.  After that fix, three months later, I downloaded again the new image for open source container  trying to do Zip and refresh and Zip and refresh fails because the URL in the repository was  wrong. And I was like, what is going on? We have packages coming from nowhere. We have  broken repositories. How people and why I am the only one that finds those stuff? Like  there was no bug report. So after those, I started looking how I can test it. So a lot  of people how I can test containers in OpenQA. And the feedback I got is that basically you  have to install OpenQA first, and then you have to spawn a virtual machine inside this  virtual machine to run in a container. Which looks like too much. Like it's way more things  to be done in that case. And it's really, really slow compared to the typical testing  of containers. For those who are interested, since we are in the university here, there  is a Google summer of code project for that. That people and students are interested to  help us and contribute to OpenQA by investigating a docker or a container backend in that case.  So feel free to check this. And so I talked with the guys that we do testing. I saw also  this and then I realized that, you know, we actually push open-source images one year ago  with no testing them. So I was like, let's try to see how I can test them. That was the  motivation behind the whole thing. So the project tries to answer how many packages  might be problematic without my knowledge. I mean, I was just lucky when I did the Zipper  search on that specific package, which was Kerberos 5 mini in that case. But can this  be that there are also other packages that are broken that I'm not aware of? And if so,  how do we know unless I test them all in that case? How many packages are supported in our  open-source container images? Are the same packages that we support in Tumbleweed, all  of them supported also in open-source containers? I mean, if I open a bug, would be a legitimate  bug or they would say, no, it's containers. We don't really support it there. And how  big undertaking is going to be this, to test everything? How much time I'm going to take  for that? Since it's a community-driven project, it's just for fun, learning, experimenting.  So I would like not to spend that much of time. How much resources I'm going to need  for that? How difficult is going to be? And how accurate the results are going to be?  That was some first questions that I had. Okay? So expectations. The expectations I  had is like, this thing has to be fast. What fast means? Fast, definition of fast for me,  is in that case that I'm going to test the whole ecosystem of open-source packages. This  means roughly 50,000 packages. I need to test individually 50,000 packages. So in that case,  I would say a week would be the maximum for me. I wouldn't wait more than a week to see  a result, five days or something like that. Trust. Trust means if I find something that  is broken in the container, I expect that the same behavior happens in your laptop.  If this is not true, my whole project is...  No, it's working. Okay, perfect.  Okay. Yes, I was saying that the whole project makes sense only if what I find is real and  it's a bug officially and we have to fix it. Otherwise, if I see different behavior in  a container that I've done in a laptop, on a desktop, it doesn't make sense to do it.  And it has to be simple. Simple meaning anybody that wants to contribute, anybody who's not  familiar with containers but can jump into it really fast without much knowledge. Requirements  for the test environment in that case. I had to build an environment in order to test that.  So infrastructure as a code. I needed something that no matter if I'm sitting in my laptop  in my workstation at work, if I visit my friend that has another distribution, I would like  not to spend time installing and configuring from scratch and messing up his system. So  everything, the whole project is containerized. You just get cloned, run one command, you  don't infect your system with any of the things I'm doing. It should be low overhead, meaning  like it should be able to scale. If I run it in my laptop, which is a dual core with  hyperthreading and 4 gigs of RAM, I should be able to run the whole project in my laptop.  And of course, if I move the project into a different infrastructure which is bigger,  20 cores, more RAM, the project should be able to take advantage of this and be faster.  Right? And then I needed isolation for exactly what I told you before. Like, I don't want  to mess my host system when I'm doing testing in that case. So for isolation, what we have,  we have virtual machines that they offer great isolation. But the tooling for making this  automatically was a little bit not the easiest, let's say. As I said before, I would like  to have things really easy for that project, like one command things. And also with the  virtual machines, I realized that the project is not going to be lightweight. Because if  I need to have one package per virtual machine, this means I need to spawn 50,000 virtual  machines. This means I will end up in 2030 in that case. T-route, yeah, quite the opposite  of what the virtual machine is, terrible isolation, terrible. So forget this. And I ended up  with container with something in between. They offer, okay, isolation, the container  can see, the kernel of your system can access the hardware resources, but at the same time  is not going to mess my host or packages in that case, only touch the file system of the  container. So I ended up with containers also because it was time for me that I was experimenting.  The methodology now here. Test, student, influence each other. What it means. Back in the day,  I was a journalist writing Linux articles and tutorials and guides. And one thing that  I really learned is that if you write a guide for somebody to follow, you have to have a  clean system always. If you are going to provide information on write documentation for somebody,  you should make sure that you download the ISO, you boot a virtual machine, you do what  you have to do, and this thing should be reproducible. So in that case, let's say that we have a  test that I have to test packets A. So we spawn a virtual machine in that case, I install  packets A. Then if I continue in this system and install packets B, it might be that packets  B has a conflict with packets A, but this is not clean anymore because packets A is  installed. It should be a virtual machine, packets A. Another virtual machine, packets  B. Something like that. So one test, the test for testing packets A should not interfere  with test of packets B. That is the scenario. Test should be ephemeral, meaning as soon  as they run, they get automatically deleted. They don't take space in my machine, they  don't use my resources anymore. I don't want to manually take care of removing containers,  their images and other logs of Docker in that case. Test should not affect the host as they  are running, what we said before. So this is a screenshot. Yeah, not the best one. I'm  using Timux here, but I can give you a demo later on when I saw you better. What I'm doing  here, basically, you see every line you see here, every of these green lines that has  success, which means open suzerox in that case, all the packages in that screen is installable,  is one container. And this in that project, here you see, for example, how much time remains  until the test is complete. Here you see file bit harvesting for logs and sending them  to Elkstack. And here I have wrote a really, really high-key parser, since I'm not aware  of any parser for zipper logs. Maybe if you are, please let me know. So I just look for  errors and I parse for specific strings in the logs to see if they are happening. So  the last time I ran that was in January. And we find, I found out that basically 98% of  our 64-bit packages we have in open suzer is super healthy and only the other percent  is not installable. The good thing with that project is that even if a package can be installed,  sometimes we have messages happening in the post installation scripts. And you can aggregate  those and see exactly how many packages are the same messages. And basically, when we  release an open suzer snapshot, if we run this after that, we maybe can say, okay, if  we fix this specific bug, 10 or 50 packages is going to be also fixed. So let me continue.  Use cases. Visualization and metrics for the installation of our packages. You can either  use the CLI and see the results on the command line, but you can also use elastic search  and Kibana and go to your release manager and give him graphs and bars and I'm sure  he will like it. Then I was like, what if open suzer had serverless infrastructure that  a user before go to Facebook, Reddit or open to the forums and start complaining that package  A is not working in his machine because he has installed a bunch of custom personal repositories  from other people, he can go something like broken forever or it's just me in that case.  So he can just, for the people that were in the previous presentation that I did, he can  just send a request with a package and he can see if this package is installable or not  and if it is in his system, it's not, then there's something wrong in his system, like  he shouldn't file a bug for that in that case. Integration in build service, I'm not sure  about that, I'm not very familiar in build service, actually I'm not at all familiar  with that. I assume that there might be a use case for that as soon as we build the  package, we can just run this outside of build service in another infrastructure in order  not to cause any overhead in the build service in that case and have the output in a database.  So this can be run also during building in that case. And this is a good entry point for  newbies like me in that case, that they want to be maintainers of some packages. When I  was volunteering for the Ubuntu bug squad, I remember that they had a panel that you  can go as a noob basically and say, you know, I want to contribute to this project, I don't  want to do translations, I want to write some code, they want to be a maintainer, but I  have no idea from where to start, I need some easy tasks. And with this project, since you  aggregate all the problems that might happen in the packages, you can classify them as  easy things to do, like trivial things to do, and somebody you can give it, you know,  take this and fix it. So there might be a collection of easy to solve problems for the  people who are passionate in Ubuntu as maintainers. And then we probably can have like a comparison.  We can have a live comparison of the status and the quality of the installation of packages  in SUSE, Fedora, Ubuntu, Debian, you name it. You can convert them and we can also check  live what version of packages they have. For example, I see people in forums say, we have  the latest GNOME, no, no, we have it, but yeah, but we have this and that. So with this  one, if you have all the logs there, you can actually take this information there, you  can have a graph and everything. So I will talk a little bit about the future plans that  I have with that and then I will give you a demo. So I will try to put this in Kubernetes  to have it in a fast way, like you have a function, you say test this package, this  package gets tested and I get the result if it is installable or not along with the log  of Zipper in that case. Really, really fast. And then I would like to experiment with that,  but now that Markus told me that he doesn't like this, you have to reconsider. Maybe with  Cata containers, Cata containers and Google device or would be the thing that I would  like to play just for experimentation. This is a fun project. Then I will try to learn  a little bit about packaging and build service and see if it makes sense to put it there  and how it works. And again, what we said before, entry point for Nubis and comparison  among other distributions. So I am basically planning to have a website for that thing  that you guys visit and you see results. So that's the presentation. Let me give you a  demo with that. So what you have to do is clone the repository. The repository, I have  already cloned this in that case. So let's go here. Let me open my browser. So here,  if you go to GitHub, I basically have a tutorial, MD for the people who would like to try that.  And also in the very front page, I have a logo though. So it's professional. You can  just go into this folder, Docker in that case. So we go to, let me zoom. And here, there  is a bunch of scripts. Let me see how much time we have. Okay. Another 10 minutes. So  you can do test it. It should be simple. Like that. It will not infect your packages. And  you put the name of the package that you would like to test. So what's happening in that  case? In that case, there is an open-suzet with container downloading in my machine,  running up, running the zipper, install Vim, as you know, with non-interactive accepting  licenses and everything because I can't interact with that. As soon as this is finished, it  will kill completely the container and remove it from my machine. But it will take the log  from zipper and store it locally. As soon as this log is present in my system, file bit,  here is going to collect it and send it to Logstas, which is a storage for enterprise  logging. Logstas will do some purely idiotic filtering that I'm doing because I don't know  any zipper or parser for that. So I'm just taking the whole thing. And it sends this  to Elasticsearch. And I have Kibana, which is the web UI for Elasticsearch in case you  would like to have some graphs and stuff like that. Perfect. So we see here that Vim is  installable right now. That's a live test in the Tumbleweed. So if I open, for example,  that's interesting, opening Vim. Yep. You see this is the complete zipper log. So in  our open-source containers, that's what will happen. Six new packages will be installed.  And here is the log of the whole thing. This now goes to Kibana, as I said. So... Yes. Here  you can see that file bit took the file, send it to Kibana. And I have here the whole log  in case I would like to have a central database or have a website, a web server that feeds  the content from that database. Okay. And I think that I have something here that you  might like. Yesterday, I ran this in my workstation at work. And I can give you some results for  the packages that they are no-arts architecture. Unfortunately, the 64-bit packages are about  19,000. And this takes about one day and a half. And since I started it yesterday, it's  not finished yet. The only thing that got finished is there are no-arts packages that  they are roughly 15,000, I think. So I'm going to take this file and put it into my project  in that case. And let me unpack it. So now we will see file bit getting crazy reading  15,000 logs. This means like since yesterday, I've managed to run 50,000 individual containers  with one package per container. So we know exactly what's going on in our open-suzet  tumbling with container images in that case. Let me, since this is sending the stuff to  the database, let's use the parser to see some stuff here in the terminal. So you see  here the timeout. I was a little bit too happy about that. And I was like, okay, I have six  core machine at work. Maybe I can run 200 containers at the same time. Yes, I can. But  what happened is that my internet connection is not that fast. So basically, I run into  the problem that I have a hard time out that I don't want to wait forever for a package.  There might be packages, if we check those packages because the open-suzet container  image is pretty, pretty small, there might be that if you try to install one, it will  try to install three gigabytes of packages. Theo loves, I know he loves. So what happened  is, yeah, internet connection problem. So just some things here. We see that some packages  here could not be able to install. So if I do like grep for that, we can see those packages.  Python Django, network manager, novel VPN, this looks like a very old one. Copana web  hub, and stuff like that, and what is also interesting is from what are installed in  that case, I have like 1,500 wrong permissions here, something like this. So if I search for  that, I get that, like setting, blah, blah, blah, trusting wrong permissions. I'm not  sure if this is a bug or not, but this is something that I get. This is not a bug. Then  why it says wrong permissions? Why there is the word wrong? I mean, yes?  Because the permissions file is a configuration.  All right. So these are not bugs that's good to know, but at least what I'm trying to say  here is that the parser checks those. In case you need them, to check them, the parser checks  those. And all those things are now in the database. Hopefully, all things are now here.  Let me refresh that. Okay. So, yes, everything is here. All the packages. Yeah. And you can  go to visualize, create a visualization, create a pie chart, and here you can have your metrics  like these are all the packages that we have, these are the packages that fail with that  error, these are the packages that fail with the other error, and so on and so forth. So  you can say, okay, classify them and give them to your user to fix them. So that's the  project so far. It was really fun for me and I received good feedback, so that's why I'm  here. Maybe it's interesting for you. You have some ideas to tell me how this can be  improved. Any questions? Yeah, let me pass you the microphone over. This can open like  this. Okay. Yes, I didn't dare to press the red button. So, whenever you submit a package  to open to the factory, there's an install check, but of course, only for that single  package and of course, it could be that it breaks another package that's already there.  So maybe these are the errors that you can find with your approach. But on the other  end, when we have like a daily tumbleweed snapshot and you say that this takes like,  you know, about a day, we cannot run this test for every tumbleweed snapshot. Do you  have a suggestion when to do this? I don't know. Maybe we can write it for Leap. I have  no idea if it makes sense. Actually, we spoke about that already in the past. You could  only test the packages which changed and you could talk to a WebMQ which is fed by the  build service, which packages are changed and you only need to test those. One thing  that your regular test would find is packages which are fair to build and are now broken  because their dependency has changed. So you would still need regular tests like once a  week or once every two weeks. But if you only want to test in the tumbleweed time frames  and test the packages where you get successful build events or publish events.  All right. Okay. We have one minute. I would just like to say that this thing works with  system dn spawn container also. So that was another experiment. You can find details how  to do it. Any other questions in the room? Yes. No. No. Okay. So no questions. Thank  you very much for being here.
openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_0_167.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_168_407.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_408_803.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_804_863.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_864_3635.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_3636_5219.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_5220_7319.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_7320_25427.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_25428_29183.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_29184_29375.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_29376_29423.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_29424_30383.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_30384_30431.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_30432_30551.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_30552_30707.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_30708_30755.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_30756_30803.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_30804_31103.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_31104_31151.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_31152_31535.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_31536_31679.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_31680_31727.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_31728_33491.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_33492_37259.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37260_37307.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37308_37379.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37380_37463.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37464_37535.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37536_37655.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37656_37715.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37716_37811.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37812_37979.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_37980_38351.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_38352_38423.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_38424_39059.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_39060_39155.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_39156_40619.png openSUSE_Conference_2018_-_Project__Egkatastasis_Ul15xYeWTAg_40620_45626.png