Continuous Integration with the Open Build Service
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bruckmayer, Christian
Getting a new Linux appliance with every commit you push sounds awesome butimpossible, right? Not with the Open Build Service, the Free Software buildand distribution system which powers openSUSE, ownCloud and Tizen! Usually alot of manual work is necessary to create your custom Linux appliance, but theOpen Build Service abstracts all the complicated technologies and makes thistask as easy as pie. In this talk, we will show you how we plug several opensource technologies together to create Linux appliances in a fully automated,continuous integration cycle. Getting a new Linux appliance with every commit you push sounds awesome butimpossible, right? Not with the Open Build Service, the Free Software buildand distribution system which powers openSUSE, ownCloud and Tizen! Usually alot of manual work is necessary to create your custom Linux appliance, but theOpen Build Service abstracts all the complicated technologies and makes thistask as easy as pie. In this talk, we will show you how we plug several opensource technologies together to create Linux appliances in a fully automated,continuous integration cycle. Starting with a commit to your favorite version control system, the OBS willbuild a ready-to-use image of an operating system for you automatically, usingthe power of KIWI. And to finish the continuous integration cycle, openQA willrun automated tests to make sure your appliance still works as expected. If you still manually build and test your Linux appliances, this talk is foryou!
 I'm not sure if it will work. I only have 20 minutes time, so in the past it took around 15-20 minutes. I'm not sure if it will finish in the end. I have a repository on GitHub created and it's a simple hello world program. So I will just change it to hello distro def room and commit it. In the end I can show you that we have a new Linux distribution created and with my package installed already. So I'm from OpenSUSE. I work at SUSE in the build solutions team so we work mostly on the OpenBuild service and some of you may notice that we brought our own beer to our booth so you can buy it at our booth. Cheers. Cheers. Anybody thirsty? Nope. So I wanted to start my presentation with some comparison to brewing beer so why it's important to have quality assurance. I tried it this morning. So what can possibly grow wrong? So if you work for a brewery and you have to be in charge of the quality of your beer then and you don't have automatic testing then you have to taste it yourself and you're constantly drunk probably. So if your beer is not good so you get bad publicity so nobody will buy your beer, nobody wants to drink it and because it's food so government is involved so they will probably give you some fine and eventually they will close your brewery which means so they will for sure close it if the beer is so bad that people get sick from drinking it. So what happens in the end? Sales decrease, costs increase and you're broke. So how is this related to software? So in software it's also important to have high quality and how do we measure quality in software? So one measurement is that the software should meet all your requirements which also means it should be free of bugs so software failers and how do we make sure that we don't introduce bugs? So I would like to decide Martin Fowler maybe some of you know him, he published some famous books so continuous integration doesn't get rid of bugs but it does make them dramatically easier to find and remove. So and this is my topic of my talk continuous integration with an open build service. So what I will talk about? This is basically the workflow I want to show you so in the right corner you see maybe some of you coding and pushing your code constantly to the upstream repository could be GitHub or some other favorite source control management system. So from the repository we want to get the code into the build service which automatically builds binaries for us so for instance RPM packages, WM packages which you can install then on your favorite distributions. And so the next step what we would like to do is build all of these packages, complete Linux image so for instance an ISO or raw image so and with my raw image I would like to be able every time I build a new package and get a new distribution I would like to be able to see if this image still works so that all the packages are installed, it boots correctly, the services are running and get some feedback in the end so yes it works or no it doesn't work anymore. So how can the build service and the tools that we use for instance we use OpenQA to test the distributions so the full stack, how can help it? The build service is so why do you need the build service? It's so you as a software developer shouldn't care about distributing your software to different distributions to different architectures you should just care about your source code and then push it to the build service and get a new package for instance for OpenSUSE, for OpenSUSE LEAP or Tumbleweed or Fedora so this is the only thing what you should care about so the build service helps here so you just push your code and your spec files or your package description to the build service and it automatically creates all the specified binaries for you so you get out different architectures, you get out different distributions and furthermore if some of your dependencies change it will rebuild all your packages so if one package relies on one other and this gets rebuilt it will rebuild the whole chain. Furthermore you can in the build service collaborate so imagine like GitHub so if you have a bug in a package you can fork it, fix it and submit it back so like you would do it with code and GitHub so that is one of our main features. So the first step is how do you get your code into the repository and you all work in free software development so I will leave out the step you all know how to get code into your GitHub or Git repository. The next step is how do we get the code from the repository to the build service. So in the build service everything is, so the main structure is you have a project and inside your project you have different packages so for instance here you have an applications project with a sub project popular and then you would have a gym or a Firefox package. So this is basically the Hello World project I showed you in the beginning and so how do we get it now into the build service so the main feature what we use here is services so services is like you know you can trigger a service to modify, to change, to update your source files and we have for instance a task service which will trigger which you can trigger and it will automatically download your source files from GitHub, repacks it and commits it to the build service so that you get automatically a new RPM package build. So there is another service for instance a set version which will automatically update your package description with the correct version. So and how do you trigger? In GitHub we have a really nice feature so there is GitHub hook already built in in GitHub so you just have to specify your token of the project and specify the there is already a default instance of the build service specified but you can enter some other so if you want to choose your own build service instance you can enter this one. So that means if you set up this with every commit you do to your GitHub or some other SCM management system and you get the source code committed to the build service which will automatically trigger a rebuild of your package. So we have the first two steps, the third step is actually automatically built in the build service so I don't need to talk much about this one because as soon as you change your sources it will trigger the rebuild anyway. But how do we get out of the packages now completely in distribution so like an ISO image or raw image? So in OpenSUSE we use KIVI which is a pearl command line tool to automatically build a fully complete Linux distribution. So we use it at SUSE to build all of our SUSE Linux enterprise and also in OpenSUSE to build all our OpenSUSE releases. So we have two versions now, we have the old version which is in pearl written and the new version which is now in Python rewritten which we use from the next OpenSUSE release on. So how does it work? Basically you have your package sources which are in the build service anyway so you order your packages and then you have an image description which will contain all the repositories or the packages you want to install. About users for instance you can specify config files to override, you can specify shell scripts to be executed while building the image. So this is basically an image description so the most important file is config.xml which is a configuration file which contains all the stuff I already mentioned. Then you have images and config shell scripts which are two shell scripts which get executed and then you have a root directory which just gets copied over your root directory and you can use it to override files. So but do you need to write all of this from scratch? No. So what we notice is that many people use SUSE Studio which is our from SUSE an online tool to create image description and images. So many people configured their image in SUSE Studio, exported all the files and imported it into the build service. So we wanted to get rid of this step so we included now the base image description so on the main page you just go to new image and then we have some pre-selection of base images you selected and then you get the pre-configured image description. So that is what we have now is step one, two, three and four. So the last step is now how do we get the, how can we boot the image, test it and check if it still does what it should do. So there is OpenQA. Some of you may went to the talk this morning from Richard so we already showed some stuff about OpenQA which is our distribution test system which we use for Tumbleweed for SUSE Linux Enterprise. So every Tumbleweed release what we do gets, before we release gets tested with OpenQA. And OpenQA is basically you have a web front end which has a database and then you have several workers which run the voice auto installer and which run different tests. So this, several back ends for instance we use, mostly we use QEMO to boot the images and run the tests and, but you can also have on real hardware via IPMI or with a lib word. So the web front end looks like this. That was yesterday evening so I tried some stuff, everything failed. So I hope that my other build will, will now finish. So how do the tests look like? So you have, it's basically written in pearl so you can write in the test everything what you can do in pearl. And we have some macros like for instance the asset screen here. And it specifies a needle which is boot loader and the time is 30 seconds. So the boot loader is a needle and it's, needle it's called because finding the needle in the haystack. And when you run the first time the test, OpenQA will stop in the, in the web UI and will ask you, I don't, I don't know this needle, do you want to create it? And then you go to the interactive mode and it shows you the output of the screen and then you can with a, with a selector you can select what do you want to see, what do you want to check on the screen? So it does basically fuzzy image comparison and checks if, if there's some major changes. So here, here you see the boot menu in, in grab and I wanted to check that there is the, the first boot option, hello for stem, still there. So I created here the needle and then it, it runs through. So that was basically about the main workflow, but what we have in the, for the continuous integration. So what is next in the, in build service we prepare actually a new release, which we will release around February or mid-February. So which will basically include the images feature, what I already showed you. So that was to, that you can create some base images, which you just need to, to adapt a little bit. And our next feature is the multiple package, so that means that you can, inside a package you can build several binaries out of it. So you don't need, it was already possible before, but you needed to, to create some links between packages and this is now built in. We use it for instance for our ARM, ARM images. So we have a base image description and then we apply several patches to the, to the different boards. And we have one package container, but inside we build many different images for the different boards like raspberry pi or banana pi. So I hope that my build is done yet. So, no, so that is, that is my build from yesterday evening, but I can show you this, this one. So here you see the test result and you see all the needles I created. So you see here, I checked for the HelloFrostam. So that is my Hello World program. And here you see my bootloader, which checks here, 100% match. And it also creates you, for instance, a video. Let me check. So you see here all the steps that got executed from, from my test. So if you, if something goes wrong and you want to compare, you just go to the page, watch the video and see, okay, what went wrong. And also if for instance a needle doesn't match, it compares, it shows you the new image and says, what is different? And then do you want to update the needle or do you want to have a different needle? Or is it, is it really a failure so that you can report it? So, so we have a booth here, which is in the building downstairs on the right. You can see my green arrow in the, in the right corner. So if you have questions, just drop by and we can show you also some more examples, how it works, how we use it for open-suit to test it. We can also show you the build service if you, and otherwise I'm basically done. So, five minutes? So yeah, if you want me to write me an email or you can find me on GitHub or Twitter. Yeah. Two questions about the source service. Does it support other conversion controls to the system? Yeah. It, it supports subversion for instance and Macorial and it's, the source service, the code is on GitHub. So if you want to have support for another system, just send a pull request and we will include it. Okay. And the same question. As far as I know, the build service usually takes a tarball from somewhere and you need to maintain the spec file within your package. Yeah. Is it possible to use the source service to download the platform or remote repository as well? I haven't tried this, but I know that for instance some colleagues do maintain their key, their key descriptions in GitHub. But I don't know if you can pull it completely so the spec file too. So. Yeah. How does it generate a version number for the package that comes from the source control? You can specify it. So there's, you saw the XML file. So that was just the base option, but you can specify more options. So you can for instance set in the Unix timestamp or the Git version or you just hard code it and say it should be 2.3 and then increment. So there are options to specify the versions. Yeah. So you can also with the key. The question was if it's possible to also build with the key other distributions than open SUSE, right? Yes. You can also build Red Hat. You can build Fedora. So but that are the only two I know so far. Not the base. Not the data packages. No. No. Sorry. Is there any control when your service is pulling new code? You spit. Yeah. It's it's specifically it's a rest RP. So you you send and say trigger service to run it and what we what we did in GitHub we included so you can send pull request to GitHub to include a service. So they have also hooks so every time when you commit to the master or some branch it will it can send a REST call an RP call and which triggers the service run. Yeah. That's that's where my question comes from. Maybe you don't want to build package every time you push something to the master. Yeah. And so in the GitHub service you can also specify the target so you can say only a master or only on this branch or only when you take some if you if you create a tag in the in the GitHub branch so that you can specify it in GitHub when you want to trigger. And are there plans to expand on platforms like the package. Why so. Debian is already supported in the build service but not in the key. So you can build the packages but you can build the image. Thank you. Thanks.
 I'm not sure if it will work.  I only have 20 minutes time, so in the past it took around 15-20 minutes.  I'm not sure if it will finish in the end.  I have a repository on GitHub created and it's a simple hello world program.  So I will just change it to hello distro def room and commit it.  In the end I can show you that we have a new Linux distribution created and with my package  installed already.  So I'm from OpenSUSE.  I work at SUSE in the build solutions team so we work mostly on the OpenBuild service  and some of you may notice that we brought our own beer to our booth so you can buy it  at our booth.  Cheers.  Cheers.  Anybody thirsty?  Nope.  So I wanted to start my presentation with some comparison to brewing beer so why it's  important to have quality assurance.  I tried it this morning.  So what can possibly grow wrong?  So if you work for a brewery and you have to be in charge of the quality of your beer  then and you don't have automatic testing then you have to taste it yourself and you're  constantly drunk probably.  So if your beer is not good so you get bad publicity so nobody will buy your beer, nobody  wants to drink it and because it's food so government is involved so they will probably  give you some fine and eventually they will close your brewery which means so they will  for sure close it if the beer is so bad that people get sick from drinking it.  So what happens in the end?  Sales decrease, costs increase and you're broke.  So how is this related to software?  So in software it's also important to have high quality and how do we measure quality  in software?  So one measurement is that the software should meet all your requirements which also means  it should be free of bugs so software failers and how do we make sure that we don't introduce  bugs?  So I would like to decide Martin Fowler maybe some of you know him, he published some famous  books so continuous integration doesn't get rid of bugs but it does make them dramatically  easier to find and remove.  So and this is my topic of my talk continuous integration with an open build service.  So what I will talk about?  This is basically the workflow I want to show you so in the right corner you see maybe some  of you coding and pushing your code constantly to the upstream repository could be GitHub  or some other favorite source control management system.  So from the repository we want to get the code into the build service which automatically  builds binaries for us so for instance RPM packages, WM packages which you can install  then on your favorite distributions.  And so the next step what we would like to do is build all of these packages, complete  Linux image so for instance an ISO or raw image so and with my raw image I would like  to be able every time I build a new package and get a new distribution I would like to  be able to see if this image still works so that all the packages are installed, it  boots correctly, the services are running and get some feedback in the end so yes it  works or no it doesn't work anymore.  So how can the build service and the tools that we use for instance we use OpenQA to  test the distributions so the full stack, how can help it?  The build service is so why do you need the build service?  It's so you as a software developer shouldn't care about distributing your software to different  distributions to different architectures you should just care about your source code and  then push it to the build service and get a new package for instance for OpenSUSE, for  OpenSUSE LEAP or Tumbleweed or Fedora so this is the only thing what you should care about  so the build service helps here so you just push your code and your spec files or your  package description to the build service and it automatically creates all the specified  binaries for you so you get out different architectures, you get out different distributions  and furthermore if some of your dependencies change it will rebuild all your packages so  if one package relies on one other and this gets rebuilt it will rebuild the whole chain.  Furthermore you can in the build service collaborate so imagine like GitHub so if you have a bug  in a package you can fork it, fix it and submit it back so like you would do it with code  and GitHub so that is one of our main features.  So the first step is how do you get your code into the repository and you all work in free  software development so I will leave out the step you all know how to get code into your  GitHub or Git repository.  The next step is how do we get the code from the repository to the build service.  So in the build service everything is, so the main structure is you have a project and  inside your project you have different packages so for instance here you have an applications  project with a sub project popular and then you would have a gym or a Firefox package.  So this is basically the Hello World project I showed you in the beginning and so how do  we get it now into the build service so the main feature what we use here is services  so services is like you know you can trigger a service to modify, to change, to update  your source files and we have for instance a task service which will trigger which you  can trigger and it will automatically download your source files from GitHub, repacks it  and commits it to the build service so that you get automatically a new RPM package build.  So there is another service for instance a set version which will automatically update  your package description with the correct version.  So and how do you trigger?  In GitHub we have a really nice feature so there is GitHub hook already built in in GitHub  so you just have to specify your token of the project and specify the there is already  a default instance of the build service specified but you can enter some other so if you want  to choose your own build service instance you can enter this one.  So that means if you set up this with every commit you do to your GitHub or some other  SCM management system and you get the source code committed to the build service which  will automatically trigger a rebuild of your package.  So we have the first two steps, the third step is actually automatically built in the  build service so I don't need to talk much about this one because as soon as you change  your sources it will trigger the rebuild anyway.  But how do we get out of the packages now completely in distribution so like an ISO image or raw  image?  So in OpenSUSE we use KIVI which is a pearl command line tool to automatically build a  fully complete Linux distribution.  So we use it at SUSE to build all of our SUSE Linux enterprise and also in OpenSUSE to  build all our OpenSUSE releases.  So we have two versions now, we have the old version which is in pearl written and the  new version which is now in Python rewritten which we use from the next OpenSUSE release  on.  So how does it work?  Basically you have your package sources which are in the build service anyway so you order  your packages and then you have an image description which will contain all the repositories or  the packages you want to install.  About users for instance you can specify config files to override, you can specify shell scripts  to be executed while building the image.  So this is basically an image description so the most important file is config.xml which  is a configuration file which contains all the stuff I already mentioned.  Then you have images and config shell scripts which are two shell scripts which get executed  and then you have a root directory which just gets copied over your root directory and you  can use it to override files.  So but do you need to write all of this from scratch?  No.  So what we notice is that many people use SUSE Studio which is our from SUSE an online  tool to create image description and images.  So many people configured their image in SUSE Studio, exported all the files and imported  it into the build service.  So we wanted to get rid of this step so we included now the base image description so  on the main page you just go to new image and then we have some pre-selection of base  images you selected and then you get the pre-configured image description.  So that is what we have now is step one, two, three and four.  So the last step is now how do we get the, how can we boot the image, test it and check  if it still does what it should do.  So there is OpenQA.  Some of you may went to the talk this morning from Richard so we already showed some stuff  about OpenQA which is our distribution test system which we use for Tumbleweed for SUSE  Linux Enterprise.  So every Tumbleweed release what we do gets, before we release gets tested with OpenQA.  And OpenQA is basically you have a web front end which has a database and then you have  several workers which run the voice auto installer and which run different tests.  So this, several back ends for instance we use, mostly we use QEMO to boot the images  and run the tests and, but you can also have on real hardware via IPMI or with a lib word.  So the web front end looks like this.  That was yesterday evening so I tried some stuff, everything failed.  So I hope that my other build will, will now finish.  So how do the tests look like?  So you have, it's basically written in pearl so you can write in the test everything what  you can do in pearl.  And we have some macros like for instance the asset screen here.  And it specifies a needle which is boot loader and the time is 30 seconds.  So the boot loader is a needle and it's, needle it's called because finding the needle in  the haystack.  And when you run the first time the test, OpenQA will stop in the, in the web UI and  will ask you, I don't, I don't know this needle, do you want to create it?  And then you go to the interactive mode and it shows you the output of the screen and  then you can with a, with a selector you can select what do you want to see, what do you  want to check on the screen?  So it does basically fuzzy image comparison and checks if, if there's some major changes.  So here, here you see the boot menu in, in grab and I wanted to check that there is the,  the first boot option, hello for stem, still there.  So I created here the needle and then it, it runs through.  So that was basically about the main workflow, but what we have in the, for the continuous  integration.  So what is next in the, in build service we prepare actually a new release, which we will  release around February or mid-February.  So which will basically include the images feature, what I already showed you.  So that was to, that you can create some base images, which you just need to, to adapt  a little bit.  And our next feature is the multiple package, so that means that you can, inside a package  you can build several binaries out of it.  So you don't need, it was already possible before, but you needed to, to create some  links between packages and this is now built in.  We use it for instance for our ARM, ARM images.  So we have a base image description and then we apply several patches to the, to the different  boards.  And we have one package container, but inside we build many different images for the different  boards like raspberry pi or banana pi.  So I hope that my build is done yet.  So, no, so that is, that is my build from yesterday evening, but I can show you this,  this one.  So here you see the test result and you see all the needles I created.  So you see here, I checked for the HelloFrostam.  So that is my Hello World program.  And here you see my bootloader, which checks here, 100% match.  And it also creates you, for instance, a video.  Let me check.  So you see here all the steps that got executed from, from my test.  So if you, if something goes wrong and you want to compare, you just go to the page,  watch the video and see, okay, what went wrong.  And also if for instance a needle doesn't match, it compares, it shows you the new image  and says, what is different?  And then do you want to update the needle or do you want to have a different needle?  Or is it, is it really a failure so that you can report it?  So, so we have a booth here, which is in the  building downstairs on the right.  You can see my green arrow in the, in the right corner.  So if you have questions, just drop by and we can show you also some more examples, how  it works, how we use it for open-suit to test it.  We can also show you the build service if you, and otherwise I'm basically done.  So, five minutes?  So yeah, if you want me to write me an email or you can find me on GitHub or Twitter.  Yeah.  Two questions about the source service.  Does it support other conversion controls to the system?  Yeah.  It, it supports subversion for instance and Macorial and it's, the source service, the  code is on GitHub.  So if you want to have support for another system, just send a pull request and we will  include it.  Okay.  And the same question.  As far as I know, the build service usually takes a  tarball from somewhere and you need to maintain the spec file within your package.  Yeah.  Is it possible to use the source service to download the  platform or remote repository as well?  I haven't tried this, but I know that for instance some  colleagues do maintain their key, their key descriptions in GitHub.  But I don't know if you can pull it completely so the spec file too.  So.  Yeah.  How does it generate a version number for the package that comes from the source control?  You can specify it.  So there's, you saw the XML file.  So that was just the base option, but you can specify more options.  So you can for instance set in the Unix timestamp or the Git version or you just hard  code it and say it should be 2.3 and then increment.  So there are options to specify the versions.  Yeah.  So you can also with the key.  The question was if it's possible to also build with the key other distributions than  open SUSE, right?  Yes.  You can also build Red Hat.  You can build Fedora.  So but that are the only two I know so far.  Not the base.  Not the data packages.  No.  No.  Sorry.  Is there any control when your service is pulling new code?  You spit.  Yeah.  It's it's specifically it's a rest RP.  So you you send and say trigger service to run it and what we what we did in GitHub we  included so you can send pull request to GitHub to include a service.  So they have also hooks so every time when you commit to the master or some branch it  will it can send a REST call an RP call and which triggers the service run.  Yeah.  That's that's where my question comes from.  Maybe you don't want to build package every time you push something to the master.  Yeah.  And so in the GitHub service you can also specify the target so you can say only a master  or only on this branch or only when you take some if you if you create a tag in the in  the GitHub branch so that you can specify it in GitHub when you want to trigger.  And are there plans to expand on platforms like the package.  Why so.  Debian is already supported in the build service but not in the key.  So you can build the packages but you can build the image.  Thank you.  Thanks.
Continuous_Integration_with_the_Open_Build_Service_0_131.png Continuous_Integration_with_the_Open_Build_Service_132_1463.png Continuous_Integration_with_the_Open_Build_Service_1464_1511.png Continuous_Integration_with_the_Open_Build_Service_1512_1583.png Continuous_Integration_with_the_Open_Build_Service_1584_2327.png Continuous_Integration_with_the_Open_Build_Service_2328_5279.png Continuous_Integration_with_the_Open_Build_Service_5280_7811.png Continuous_Integration_with_the_Open_Build_Service_7812_8579.png Continuous_Integration_with_the_Open_Build_Service_8580_8627.png Continuous_Integration_with_the_Open_Build_Service_8628_9563.png Continuous_Integration_with_the_Open_Build_Service_9564_10043.png Continuous_Integration_with_the_Open_Build_Service_10044_10631.png Continuous_Integration_with_the_Open_Build_Service_10632_11171.png Continuous_Integration_with_the_Open_Build_Service_11172_11363.png Continuous_Integration_with_the_Open_Build_Service_11364_13739.png Continuous_Integration_with_the_Open_Build_Service_13740_14495.png Continuous_Integration_with_the_Open_Build_Service_14496_16175.png Continuous_Integration_with_the_Open_Build_Service_16176_17039.png Continuous_Integration_with_the_Open_Build_Service_17040_17699.png Continuous_Integration_with_the_Open_Build_Service_17700_20147.png Continuous_Integration_with_the_Open_Build_Service_20148_21431.png Continuous_Integration_with_the_Open_Build_Service_21432_22367.png Continuous_Integration_with_the_Open_Build_Service_22368_24311.png Continuous_Integration_with_the_Open_Build_Service_24312_24491.png Continuous_Integration_with_the_Open_Build_Service_24492_25091.png Continuous_Integration_with_the_Open_Build_Service_25092_25235.png Continuous_Integration_with_the_Open_Build_Service_25236_25283.png Continuous_Integration_with_the_Open_Build_Service_25284_25427.png Continuous_Integration_with_the_Open_Build_Service_25428_25535.png Continuous_Integration_with_the_Open_Build_Service_25536_25895.png Continuous_Integration_with_the_Open_Build_Service_25896_26963.png Continuous_Integration_with_the_Open_Build_Service_26964_34655.png Continuous_Integration_with_the_Open_Build_Service_34656_34715.png Continuous_Integration_with_the_Open_Build_Service_34716_34778.png