Relax-and-Recover Automated Testing
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
D'haese, Gratien
Relax-and-Recover Automated Testing is a sub-project of Relax-and-Recover to perform fully automated recovery tests without human intervention. This way we can quickly verify every unstable release and test each commit as soon as it fits us.
 Good afternoon everybody. Welcome to this talk, Relax and Recover Automated Testing. First of all, are there Relax and Recover users over here? Some of them? So not yet. Hope you all will convince you after this talk. So it's always good. This is the agenda that I will go over today. I will introduce myself. I will give you a short introduction into what is Relax and Recover. And of course the main topic will be the Automating Testing followed by a demonstration. So it's always nicer to see it than just to have to place slides. The slides anyhow will be available on the Froskem site and will contain also the more or less the Automated Testing step by step so that at least you have some feeling of what the Automated Testing is. And of course you can do it yourself because it's super sourced. But I come back to that later. First of all, where am I? I'm already an old guy who's a very long time in the Unix. And also in Linux from the 90s. I think it was one of the first in Belgium that was using Linux. And I've been in RIR from 2006 but before RIR there was a makeshift drum recovery. Maybe some people have heard of that. That was the previous project that came before Relax and Recover. That was also in Linux Disaster Recovery Project. I have some other projects of course but you can find it on my Hitchhiker page and on my website. Okay Disaster Recovery. So there's some basics. It's a very short basic introduction. It is about you lost your hardware by fire, the crash, whatever. It can be also a virus attack for example. Anything that disturbs your business can be in the category of disaster. Furthermore you also have business continuity. That's going much farther. It's much broader. It's about your business as out of business. You have to build up your data center, your telecommunications. You have big plans around it. And Disaster Recovery is only a tiny part of it. So to differentiate what is Disaster Recovery. For techies like us it's more like our computer or data center. Key components like a database server, mail server to restore that as soon as possible. But business continuity is about the business and business is always central because that's the one who pays us check at the end of the month. So that is the last picture about business continuity. And that is where you can see very clearly that Relection Recovery really fits. It's in the recovery part of code. But you have a lot of steps in business continuity. Sometimes I give a talk about business continuity itself. It's also a very interesting topic to talk about. And it is something that this never stops. Because your business is changing always. You introduce new tools, new business plans, new integrations of new buy-ins and buy-outs and whatever. So it always changes. So you always have to rehearse, rethink and re-exercise. Also true for your Disaster Recovery you have to make exercises. Don't blindly trust that the tool like Relection Recovery will work out of the box without testing. Very important tool to remember. Testing. Okay. That is something I probably don't have to explain to you what this Disaster Recovery is. It's not the question can it happen? It's just the question when will it happen? And it does happen to everybody of us. It does happen to me. Even I am an expert to it. So always have the backup plans ready. You can make your life easier by having a good inventory of your hardware, for example, of your software. Central database, whatever that catalyze everything and make plans of it. What is there? Because when you lose it and you have no clue what is in your business because your business can be worldwide, can be hundreds of different smaller sites. If you lose one of those sites and you have new clue what's in it, okay, that's very difficult to rebuild. So you have to make plans of that. But Relection Recovery can help you with a tiny part, your computer part of, in this case, Linux only. But it does cover different architectures. It does cover PowerPC, Intel, even Ethereum. It's still possible. All right. Let's continue. Still frequently, people tell to me I have a very good backup solution. And that's enough for me. It's only good for your data. It's not good for your overall business to be online again. So it has to be smaller very quickly. And this is maybe the best light to explain to your management why backups are not enough. Backups are good for your data. But to do in data recovery from scratch, then you have some good other tools needed for doing that. And Relection Recovery is really fitting in that piece. It can shrink your recovery time very drastically. As you can see, if you do it from the first line, you have the OS that you have to reinstall. That takes time. You can tell me you're using the cloud, but even in the clouds you can lose your your conference management tools because conference management does not cover everything because I'm daily using it. So there's still caps in it that you don't cover with Conferenced Management, that you forget about it. That's not ready yet. So therefore, Relection Recovery can help you to have your OS and everything that's in it, your Conferenced Operator System can be covered with that. And to restore that is much, much easier and much faster at the end, even with Conferenced Management. And we do practice it on the companies that I work for. So I hope that I make myself clear that Relection Recovery is about disaster recovery of your Linux Operator System. And basically the tool itself is completely written in the BAS language. Why we did, we chose BAS? Because every Linux system administrator is supposed to understand and use BAS. And that in practice that is true. So we therefore we chose BAS. It's a very modular system. That means that you can write a very tiny script and put it somewhere in the tree and it will be picked up automatically. So there's nothing that you have to do especially you don't have to dig very deep in all the sources. If you know the concept of Relection Recovery it can be used for to add something yourself. By the way, I have several other open source projects which are based on the framework. And the framework that we used was was initially created by Schleumer Schapiro which is my colleague from Berlin. And that is still used in rear and other processes as well. So it's not only dedicated for rear but it is so easy to to plug in something in there. And rear is meant that you configure it on your system or your systems if they are similar and you install it, configure it, test it and forget about it. On a weekly basis you can have a new snapshot of your system disk and you're good to go. Furthermore, it integrates with many many different backup solutions. So the basic rear as it was created in the beginning it was effect using tar and make the tar of your operating system and put it somewhere on an NFS file certainly. And when you need it you can boot from the ISO and you could restore it from that point in time. But furthermore in time we also did integrations in many open source backup solutions but also commercial solutions like Tivoli from IBM, EMC Networker, Data Protector and Barrios is in there, Bacola is there and many others that I forget about. But okay, it makes it easy. So the the main point of relax recovery is to make your task for disaster recovery as easy as possible and design it in that way that it can be configured in a flexible way that fits your needs. All right, maybe that sounds maybe simple but okay. How does it work? Of course your your operating system consists out of disks so the basic thing is that it has to make a layout, save the layout of your disk environment, your logical volumes, your mirror disk, your hardware rates, software rates, whatever. And all your file systems, how it is labeled, is all written into a configuration file and that is kept on your life system but also on your recovery system. And also the bootloaders that you're using, the major bootloaders that are in use. Furthermore, then it stores the files. If you request it for it, if you use say an external commercial backup solution, then it only stores your basic system layout and when you restore it, it recreates the disks and then we restore from the backup and that's part of the integration. And also very important, create the bootloader, the boot system that you use to recover from it. And all that can be done and is done online. So there's no offline thing needed. It's all you can just install it on predictor system, configure it and it will work. Never use it on the prediction system without prior testing in a development environment of course, similar to your production. Never trust the tool as such. Even I say it's a good tool but never trust it. Always test it. The recovery, basically it can, the recovery system can be made on a nice environment. It can be on a network via pixie booting or it can be on a new USB disk. That are the three main recovery possibilities that are in practice today. Besides the recovery operator system, which is in fact from your live operator system, it's the same kernel that we're using. So that's always nice. We copy a lot of other tools that you may require to do some rescue environment. You can use the rescue image to do file systems and repairs for example, not only to recreate everything from scratch. That's not always the needed but okay. So therefore we copy a lot of tools. And everything, your complete operating system or rescue operating system is in RAM. So you never touch your live disk unless you decide to do it to do a recover. Of course your live disks will be overwritten. So this is a graphical overview of the backup flow and output flow. If you look in the configuration files of relaxant recovery you will see a backup definition and an output definition. The output definition is for the rescue image. So where you want your rescue image to be, do you want it to be an ISO and the location where you want to store it. And the backup is a fact about your operating system. If you do a backup netafacilist for the main default one, that's for the tar or the r-sync internally. But if you want to, for example, using a Tivoli backup, then you define backup equals TSM for example, or DP for data protector. Then you're saying to Rear, okay, your data of your operating system is in another backup solution. So we don't use the internal one. And the internal one is effect tar or r-sync. So that is the main differences. And in backup you see the internal tar r-sync and the external ones can be any kind of open source or commercial backup solution. It also integrates, for example, with duplicity. This is also a nice cloud backup solution. The uses of Rear, you have effect to three main parts. You have to make rescue. That's to make your rescue image only, not backup, only the rescue image. That's its effect. If you're using an integrative of a commercial or an open source backup solution, it only requires a make rescue because you don't need to make backup because make backup is meaningless at that point. Make backup is meaningful if you're using an internal backup solution, like tar, r-sync, whatever. You can also run make backup only. There are many others. But I would say make backup and the recover of the two main options within Rear to do one and make backup is to make a rescue image including the backup. And the recover is effect if you have a crash and you want to recover from the backup that was made prior. And as already said, before you can easily integrate it with, for example, config to HTML with a find quite handy to have a configuration management overview in text or HTML included on the ISO, USB, or on the network environment. So you can always look what was the configuration of my system before I need to restore. Also very handy if you want to do cloning, for example, because Rear can be used for cloning systems and it is used for cloning systems with a lot of comparations. So you want to get started. These are the main pointers to the website. Also to the RPMs net solution because we haven't daily built of Rear with the OpenSuzi built services. This is a very helpful tool that OpenSuzi and Susi provides to the developers. So anything that we, once we do and check in into our hit repository, then my server will detect it on a nightly basis and will automatically rebuild and push it to the OpenSuzi build service. And the next day you have it available or a few minutes later. On the different architectures for Dabion, Ubuntu, Fedora, Sless, REL, Sentos, etc. etc. Furthermore, the official source is on a jitgup, like I mentioned already. Also the issues are there. If you have an issue with it you can report it over there. And maybe you know it or you don't know it. Rear has been added to the base operating system of Red Hat. So you will find in the 7374 still the Rear version 172 with I think it's the 9th subversion that they created with big fixes that were in our issues. But in the 7.4 that has been released in beginning of August this month, it's rebased to the Rear version 2.0. And that is a major step from the Rear 1 versions to the 2 because we had it in the re-numbering. And also a lot of new features and stability was included in Rear. So if you're still using a Rear 1 version, it's better to go to the Rear 2 because it's more stable, it's much more tested also with automated testing, for example. But okay, this one to say it's in the official repositories of Red Hat and also Sless has the more of less official repositories in there thing. One of our main developers is also working for Sless, I'm for Susan. So that helps of course. Right, I'm coming to the topic of today in fact, it's testing Rear. Because Rear is already more than 10 years old and in the beginning we were using physical hardware, a lot of virtual machines to test it, but it is very cumbersome because you have made some modifications, you get new features from users who want to integrate their backup solution, for example, to give an example. And we have to blindly trust that it works. And of course we are all developers and nobody is perfect. I'm also not perfect of course. We make mistakes and sometimes mistakes are integrated or checked in automatically without being tested properly. And it's in fact almost impossible to test everything because like I said in the beginning Rear is working mainly built for Intel, but now the IBM is pushing hard. One of the developers is also now from IBM for the PPC and PPCLE46 environment. So it's growing fast in that community. Of course I don't have a lot of, I don't have any PPC system at home, so it's for me difficult to test on that kind of hardware. So therefore this environment was also built and IBM is also trying to integrate Rear and made the testing to test their new features in it. So it's just too broad to test everything. We do our best to test the basic stuff. The internal backup solution is tested. I can test the ISO, I can test the PPC booting. So that's convenient and also for example for Barrios, if our Barrios uses in here, that's the open source backup solution, that's the integration they asked me to to help them to do the integration testing with Barrios. And that's also done with the automated testing tools. Like I said, it will be a challenge to test everything, but we try to do our best. Rear Automated Testing Effect is an open source project, make it very clear, but it was initially written for customers who paid for it. Because we're doing more than 20 years open source projects for free and still have to live as an independent consult. So it's nice to get paid sometimes. So doing the automated testing is kind of finding a balance between continuous integration and automated testing. Which is not easy if people here are in DevOps environment that they probably know. It's always a battle. And even here it's a battle. But like I already mentioned, the open source build service is a very good first step. Because that provides me with an automated way of building the RPMs and they are available. Which is nice. So I don't have to do it myself every day for different platforms. Which is good. It helps for developers. And we do get feedback from a lot of developers. It's on operate systems which I do not have or do not use. They come back with features, enhancements, with bug fixes and also with bugs and forms. And we do have developers which are game helping us and left. And we some have a few developers who are staying already on board for several years. Which is nice. We have now five key developers in our team. Which is good. Like I said from Suzy, we have a very good one. From IBM. We have Slomo Shapir myself. And we have also a guy from HPE from Slovenia. Who is also a very excellent person which is helping us. We have some previous very good ones. Which are now left the team but still in the background sometimes. But okay. That's open source. It's coming and going. And we are always lucky to have some good people on board. And if they lose their focus and do something else, that's fine. Know our feelings. We are glad with all the help that we can get. Support is done via the main channel is Jitschup. The issues. There is a lot of free support as I said. But you can also have commercial support via my company. All right. The automated testing. We are coming closer to the demonstration. That's good. So it was initially written for the customers who have a support contract or a subscription contract. And currently the first time that I talked about the automated testing was of Fosden. Beginning this year. And then I only had it for CentOS 7. And on the LIFERT. So I gave a demonstration with my Linux laptop with LIFERT. KVM LIFERT. But now we have Ubuntu. And to have it more convenient. Also integrated into the Oracle VirtualBox. And I will give a demonstration with my Mac. Just to prove that it works on a non-native Linux system as well. So I have a different operating systems at home. I have using an own Linux, KVM Linux with Oracle VirtualBox. But here it will using my Mac system. Just to prove it. The next slide will show you the overview of the setup. But the first machines that I use are provisioned automatically with Ansible. In February this year at Fosden it was still using a script. And also gave a talk at CBIT. And then somebody in the audience talked to me and said, why don't you do it with Ansible or something else. That's a very good idea. Because I am now also on DevOps. And it was also a good exercise for myself to dig into Ansible and to do the complete provision of the virtual machines. That means what do I do with the operating system is a box. So updating it, installing rear, of course, all the dependencies of rear will be installed automatically. But for example, if I am using barriers as an integrated test, I have to install barriers. On the server side I have to configure it. You have to become a barrier server. And the client has to be configured. So these things are all automatically done. And it's done for Ubuntu as well. Because they are always between the different versions, different dependencies. It's a bit awkward, but that's life, of course. So that's all covered by Ansible. And this demonstrates I will not do the provision from scratch because that will take 10 minutes. I will just demonstrate the admitted testing itself. But if there is time, I know later on I can always show you that provisioning is working with Ansible. So in this setup, what it will do is automatically it will launch a client and a server virtual machine. It will automatically create a disaster recovery image from the client, make a backup to the server. If I'm using Oracle virtual machines, virtual box in this case, that means booting needs to be done from the host system. If I'm using, for example, KVM live-word, I can boot from my server VM, which is not possible from virtual box. But that's the limitation of virtual box. So it doesn't mean just a different way of starting up your recovery image. But that means that the client will, in this case, this demonstrates I will be using the internal network, netFS solution, and I'm using tar to make a backup of my client system to the server VM. But the same can be done with Barrios. If time needs, I can always demonstrate it. In this case, I will boot via Pixi, but can also boot via ISO. When the backup is completely made, I will stop the client VM and automatically start the recovery VM. The recovery VM will automatically do a restore from scratch. It will recreate your disk layout, restore everything from scratch. And that is done. It will reboot and your client VM will be online again. So that's the basically of automated testing. This is the graphical overview. On the top side, you will see the host system. And I call it the vagrant host because vagrant is used to drive everything. And on the left side, you see the client in the middle of the server, and there are the recovery VM. So there are three VMs created. The client and server are already provisioned. I did it on front. So they are containing everything. But, okay, that every time that I start, you admit testing system, I do not reprovision everything for the same operating system. If I want to use CentOS and it's provisioned, and I want to test a new version of my relaxation recovery version, I do not to reinstall the operating system from scratch. That's not needed. I just have to update the relaxation recover version effect and restart the backup. So the reprovisioning is not required every time. It's only required when I want to switch from CentOS to Ubuntu because that's kind of logically. You can destroy your box and recreate it automatically. That's done automatically. So in different ways. So you have an internal network. It's the 33 network, which I'm using to do the backup internally between the VMs. And you also have an external network, and the HCP network, which I can talk to the internet. If you want to download the latest relaxation recovery version from open source build services, it's going to the internet to download it. And you will see it in the demonstration. All right. And to boot the recover with Pixi, it boosts from the hypervisor. If I'm using LipFort, it's going that way. So that's the difference between KVM LipFort and the virtual box situation. This Mac system is using the virtual box. So I'm booting from the hypervisor. But okay, you don't see the difference effect. And that's all covered by the script of the automated testing. It's also a script. Automated testing. It's written in BAS. It's open source. You can find it. I will show you the pointer later. But to test the environment, I'm using Vagrant. That's the main feature. Vagrant and virtual box of KVM. If you're using KVM, then you also need the LipFort plugin. And HIT is also required to download the latest version of the automated testing software and to clone the Vagrant files, because the Vagrant files are included in the project. I can show you. I will demonstrate it anyhow. So to conclude before I dig into the demonstration, it's open source. Everybody can use it, test it out and give feedback to me or to the project. What is missing, maybe prefixes, whatever. It's always welcome. New code is written only for customers, in fact. It's not doing it for free. Unless it's a serious work, of course. You can clone it. This is the address. You just have to go into the directory and type minus gates for the help. The automated test is the main script. There's only one script effect. And the rest is sensible. Playbooks. By default, you can use the Vagrant user to log in. You can also use the root user to log in. And the password for both is Vagrant. That's also auto provisioned by the Ansible scripts, playbooks, re-scipes, whatever. As said, there are different ways to log in. You can use the Vagrant subsystem to log in to your VMs. Or you can use secure shell. Or even you can use VNC. Or it's a virtual box. You can use the console of the virtual box, of course. By default, there are also three ports generated. That's for VNC that you can connect to the client, server, or the recover virtual machine. Right. You want to try it yourself? That's in fact minus p is the provisioner. If you're using KVM, livevert, then you have to give livevert. If you're using virtual box, the default is virtual box. Then you don't have to give any option at all. All right. There is also a testing part. But the testing part which is existing today is effect and donation from Red Hat using BeakerLip. And these scripts are written in the BeakerLip language effect. They're using that tool, if I had the Red Hat testing system, whilst there are main system BeakerLip, to test internally the stability of relax and recover. And that is still used at Red Hat. But as we are using more and more the automated testing system for other operating systems, like vendors, like Debian, Ubuntu, Slash. So I'm thinking very hard to replace these scripts by another system like a bus automated testing system. But that's for later. I can't demonstrate it. It's working because Red Hat is using it and it's integrated in this tool with the minus t option. Well, it proves the fact the stability effect of your OS, whatever you want to test. And now the most important stuff we give in demonstration. Is it visible? Have to go up a bit, maybe? All right. Here you can see the main script. It's over here. The minus h is the help effect. And it shows you different options like the minus d. It's the distribution. The default is center seven. And that was the first customer who asked for it. But now there are two others, like the Ubuntu 14 and the Ubuntu 16. You just have to mention it. If you want to know, okay, how do I know? Just do an ls. And you will see here the Ubuntu, these are the effect of the operating systems. These are the test scripts. These are the internal rearscripts. These are the test stuff. And the templates is effect the, it's always interesting to look at it. These are the configuration files which are used internally for testing here. You can see this is the ISO booting with netfs and fs. This is the booting with URL type. This is default. And this is for BIOS. BIOS def names is for the Ubuntu. And this is with ISO for CentOS. That will change in one of the upcoming rears versions. I will do auto detection of BIOS def names. But okay, going too deep maybe. That's if you're using SAP or these kind of hardware, then most of the time you're using BIOS def names. It's not eaten at zero, but it's the real physical hardware names that you will see. And in this case, Ubuntu was using the physical hardware and in the recovery it was the aliasing effect. So it was a bit awkward. I made a different configuration file for it. But okay, it doesn't matter. Right, for this test, we will be using, let's see, still have time. Yes, we will be using CentOS 7. So that's the default. Using Pixi, it's also the default. First box is the default. So I don't have to give any parameter here. I can just announce it. Let's announce it. So it tells me CentOS 7 will start up Vagrant, the providers, and then second the client. It tells me there's already an update for the box itself. Okay, that's if I have to do box inload for that. So that's also useful to replace the box. And if you have a new box, then I have to do a reprovisioning, of course. But anyhow, it doesn't matter for the moment. It will start up the client. It will start up the server. Very soon you will see the server coming up. There is the server. Again, the same kind of text will appearing. So this is pure Vagrant. Vagrant is driving the startup of your boxes. And it's using internally in the back end, virtual box. Okay. I always find it nicer to demonstrate something. And then it's easier to understand the complete picture. So now, yep, the client and server are online. Just five seconds delay. Just if I want to, was mistaking, control C. If you do it now, it will destroy the boxes. And if the boxes are destroyed, then you have to do reprovisioning. And that takes more than 10 minutes. So hands off. Check if the networks are online. That ping test is okay. That means that the client and server can talk to each other. Of course, if you want to back up. Here it is. Yep, there's already a new rear uploaded. It goes very fast. Sorry for that. This is the configuration file that is being used. That is coming from the templates. It's copied automatically. Every time you start, it's automatically recopied to the client. And it's already making a backup. And you can see here, it's the first from the 7th of August that is being used. Relics and recover. It's making a snapshot of your bootloader file systems. And now it's already making a backup to the client environment. Which is in this case, automatically, because it's designed to sit that the backup is going to the TF2P area of your virtual hypervisor effect. You have to do nothing for it. It is built into the tool that it knows which kind of hypervisor is being used. That takes only a few minutes. Is there something more interesting to know maybe here? Automatically add the kernel options with the net.if names equals zero. That is for using the alias names. So to disable the bias def names. This is convenient because last year I gave a workshop at the open source backup conference. And we had issues with that. I never had issues with Lippford. But most of the students were using virtual bugs that problems with the alias names of your interfaces. But if you add this to the kernel options, then it goes away. So you see backup. Backup URL. Okay, it's done. Backup was successful, it says. Okay, the backup client area will be on the pixie boot area will be made readable. I'm halting the client for some machine now. And starting the recovery machine. I'm using another box. This is an empty box. Again, it's the same vacant. It's driving, driving the complete stuff. And now this is virtual box taking over. You will see it's pixie booting. So it's really live booting. No cheating. It will, because it's virtual box, the first network will not work. But this one, the second one will respond. And then it will load automatically your kernel and your initial renders, your rescue image from relaxation recover. It will boot it up. You can see it will reconfigure the network. And it's already starting with reformatting your internal disk. So your disk layout and automatically the restore of your data or operator system will be restored automatically. That goes quite quickly. Afterwards, you will also see that it will rebuild the kernels. Not the kernel, of course, but the initial rend disk. Because it's another size of disk. So it has to see it's another box. That means internally it's another system. So what I'm doing here is effect kind of cloning. I'm cloning the client on another fresh hardware. And you can see it here. For every version of your kernel, it will rebuild your initial rend disk of that operator system, the client in this case. It's just rebuilt on another system. You will see it at least twice. And it takes longer effect almost than the restore of your OS. Once that is done, you will notice that the system will reboot. And, well, then we hope, of course, if the relaxation recover does not contain any bugs, that it will come up automatically. So here is the rebooting in 30 seconds. That's also something done by relaxation recover. So that's a timing that I can make shorter or longer, even zero. But, okay. It's always nice if you see an error that you can interrupt and check your restored environment. Okay. 30 seconds takes some time. Okay. Stopping automatically the system, and it will restart. You will see automatically that, again, it will pixie boot. But the pixie boot configuration was automatically changed that it will boot from the first internal disk, not from the ISO or network anymore. So that's all done automatically in the background. And here it is booting up your client system. Takes a few seconds. And, oops, there is the prompt. You can just log in with vagrant as password. And, well, the client is there. I recovered the client VM on the recovered VM. I had to do nothing except running this script. And I can do that every day. I can do it for different backup solutions for backup. For the moment, it's only barriers. But I can do the same thing for barriers on central seven, Ubuntu 14, and Ubuntu 16. And whatever is missing can be added, of course, but preferably by sponsoring. That was my main talk. The slides that will be provided contain a fact and summary of the complete recovery steps made by the script. And also the pointers to the main site, etc., etc. So, are there any questions so far? Because this was my talk effect. No questions? Yes? What is the effect of maintaining this? Because there are new kernel versions, not a line, I think. So, how much effort goes into this? A lot of effort. You can see here, this is the issues. There are, in general, around 70, 80 issues open all the time. And we are working with, let's say, three people, not full time, but in our spare times. And also, like the person from Suzy is almost working 80% of his daytime. And the guy from IBM is working, let's say, 40, sometimes 50% of his daytime on cases. Because, like I said, in my talk, IBM is hot these days for the SAPA, SAPANA environment on PPC environments. The customers are requesting it that there is a decent, relaxed, and recovered solution for it. Disaster recovery solution, I must say. And therefore, they put a lot of effort to test it on their platform, of course. I know. Okay, so the question was, how do we keep up with the rolling new features that are coming from operating systems from new versus, et cetera, et cetera. Indeed, that's a very intensive labor. I'm using to do that for 10 years every new time. For example, Ubuntu, from going 14 to 16 major versions, they were split up from ISO Linux into Pixel Linux and what was it, USB Linux, and stuff like that. All these packets, dependencies were different, namings, so you have to foresee that. And that is, well, normally, in most cases, when there is a new version, we ask the community, common users like yourself, are testing it. And a lot of feedback are coming from guys like you that make issues and say, okay, doesn't it mint something? And it fails for that and that reason. And then we ask, so meet the log files because it really has an excellent log basis. And we can find a common solution for it. That takes time. We do have customers who have a support contract, which is much faster, of course, because we are paid to do that. I'm working for customers who are renting me for maintaining their relax and recover basis, which is for 10,000 systems, making the engineering, that goes better. But okay, it's going so fast, it's not easy to maintain everything and sometimes it takes months to get something fixed. But the main basis, let's say, for the main versions of Linux, being less red-head, Ubuntu, Debian, is working fine. That's okay. Maybe another question, because I mean, I really will drive this out and I'm really loving that the whole search, you know, that you have backup and recover and I mean, nobody wants backup. I mean, backup is, everybody wants to recover. Everybody wants to be able to recover. I mean, don't care about backup. I mean, you need to do it, but you want to be able to recover. And it's really important that you know that it works. And now, I mean, the closing subsystem in Windows, getting hopefully better, better, so we have any intention, maybe, I mean, that is basically really no such thing in the Windows. So is there any chance that this could someday also backup and recover a Windows system? Oh, the question is, can relax and recover backup a Windows system? The question is yes. Only on Laptop that contains a Linux system. It's called backup equals block loan. That's written by one of our colleagues, DeVlat from HP here from Slovenia. He wrote it and it's working fine. But okay, it's not real, it will never work natively on Windows, I think. It's a Linux software and it's running on a broad version of Linux. Different kinds of OS, different kind of hardware. We love to see the ARTS 64 hardware to being incorporated because we got already questions from Ret yet and asked them, okay, can we have some sponsoring from you? And it goes quiet. So it's always just a regular between doing it for free and doing, because if I were for real, I cannot be paid for other projects. But I love to wear for real. Another question? You said earlier, there is all the interconnection of batch screens. So why does the hardware come in very often? Should it be built for any kind of processing? The question is, rear is written in batch. Why is the differentiation in the different hardware required? Well, the question is very easy. Different hardware is using different boot technology. The PPC environment is using at least two different boot technologies, the Yaboot and Lilo. Within the Intel, you have the ultra legacy BIOS way of booting that can be used, Grip, Grip 2, Lilo, and Grip 8 these days, or UAFI. That's using a different kind of boot, Grip 2 most of the times. But even within UAFI, there are different boot technologies. So the way of booting within that area is different. So all these things are already inside rear, and that is a moving duck effect. So new technology is added. We are always coming after defects. Well, most of the times, that's okay. It's only a matter of time that it's get integrated. And that's why there are different in rear, you will see different tracks for different hardware technologies. Okay. If there are no more questions, I thank you for your time and for your interesting questions. Have a nice day. Thank you.
 Good afternoon everybody. Welcome to this talk, Relax and Recover Automated Testing.  First of all, are there Relax and Recover users over here? Some of them?  So not yet. Hope you all will convince you after this  talk. So it's always good. This is the agenda that I will go over today.  I will introduce myself. I will give you a short introduction into  what is Relax and Recover. And of course the main topic will be the  Automating Testing followed by a demonstration. So it's always  nicer to see it than just to have to place slides.  The slides anyhow will be available on the Froskem site and will contain also the  more or less the Automated Testing step by step so that at least you have some  feeling of what the Automated Testing is. And of course you can do it yourself  because it's super sourced. But I come back to that later.  First of all, where am I? I'm already an old guy who's a very long time in the Unix.  And also in Linux from the 90s. I think it was one of the first in Belgium that was  using Linux.  And I've been in RIR from 2006 but before RIR there was a makeshift drum recovery.  Maybe some people have heard of that. That was the previous project that came  before Relax and Recover. That was also in Linux Disaster Recovery Project.  I have some other projects of course but you can find it on my Hitchhiker page  and on my website. Okay Disaster Recovery. So there's some  basics. It's a very short basic introduction. It is about  you lost your hardware by fire, the crash, whatever. It can be also  a virus attack for example. Anything that disturbs your business  can be in the category of disaster. Furthermore you also have business  continuity. That's going much farther. It's much broader.  It's about your business as out of business. You have to build up your  data center, your telecommunications. You have big plans around it. And Disaster  Recovery is only a tiny part of it. So to differentiate what is Disaster Recovery.  For techies like us it's more like our computer or data center.  Key components like a database server, mail server to restore that as soon as  possible. But business continuity is about the  business and business is always central because that's the one who pays us check  at the end of the month.  So that is the last picture about business continuity. And that is where you  can see very clearly that Relection Recovery really fits.  It's in the recovery part of code. But you have a lot of  steps in business continuity. Sometimes I give a talk about business  continuity itself. It's also a very interesting topic to  talk about. And it is something that this never stops.  Because your business is changing always. You introduce new tools, new business  plans, new integrations of new buy-ins and buy-outs and whatever.  So it always changes. So you always have to rehearse, rethink  and re-exercise. Also true for your Disaster Recovery you have to make  exercises. Don't blindly trust that the tool like  Relection Recovery will work out of the box without testing.  Very important tool to remember. Testing.  Okay. That is something I probably don't have to explain to you what this  Disaster Recovery is. It's not the question can it happen? It's just the  question when will it happen? And it does happen to everybody of us.  It does happen to me. Even I am an expert to it. So  always have the backup plans ready. You can make your life easier  by having a good inventory of your hardware, for example,  of your software. Central database, whatever that catalyze everything  and make plans of it. What is there? Because when you lose it and you have no  clue what is in your business because your business can be worldwide,  can be hundreds of different smaller sites. If you lose one of those sites  and you have new clue what's in it, okay, that's very difficult to rebuild.  So you have to make plans of that. But Relection Recovery can help you  with a tiny part, your computer part of, in this case, Linux only.  But it does cover different architectures. It does cover  PowerPC, Intel, even Ethereum. It's still possible.  All right. Let's continue.  Still frequently, people tell to me I have a very good backup solution.  And that's enough for me. It's only good for your data.  It's not good for your overall business to be online again.  So it has to be smaller very quickly. And this is maybe the best  light to explain to your management why backups are not enough.  Backups are good for your data. But to do in data recovery from scratch,  then you have some good other tools needed for doing that. And Relection Recovery  is really fitting in that piece. It can shrink your  recovery time very drastically. As you can see, if you do it from the  first line, you have the OS that you have to reinstall.  That takes time. You can tell me you're using the cloud,  but even in the clouds you can lose your  your conference management tools because conference management does not cover  everything because I'm daily using it. So there's still caps in it  that you don't cover with Conferenced Management, that you forget about it.  That's not ready yet. So therefore, Relection Recovery can help you to  have your OS and everything that's in it, your  Conferenced Operator System can be covered with that.  And to restore that is much, much easier and much faster  at the end, even with Conferenced Management.  And we do practice it on the companies that I work for.  So I hope that I make myself clear that Relection Recovery is about  disaster recovery of your Linux Operator System.  And basically the tool itself is completely written in the BAS  language. Why we did, we chose BAS? Because every  Linux system administrator is supposed to understand and use BAS.  And that in practice that is true. So we therefore we chose BAS.  It's a very modular system. That means that you can write  a very tiny script and put it somewhere in the tree  and it will be picked up automatically. So there's nothing that you have to do  especially you don't have to dig very deep in all the sources.  If you know the concept of Relection Recovery it can be used for  to add something yourself. By the way, I have  several other open source projects which are based on the framework.  And the framework that we used was was initially created by Schleumer Schapiro  which is my colleague from Berlin. And that is still used in rear and other  processes as well. So it's not only dedicated for rear but  it is so easy to to plug in something in there.  And rear is meant that you configure it on your system  or your systems if they are similar and you install it, configure it, test it and  forget about it. On a weekly basis you can have a new  snapshot of your system disk and you're good to go.  Furthermore, it integrates with many many different backup solutions.  So the basic rear as it was created in the beginning  it was effect using tar and make the tar of your operating system and put it  somewhere on an NFS file certainly. And when you need it you can boot from the  ISO and you could restore it from that point in time.  But furthermore in time we also did integrations in many  open source backup solutions but also commercial solutions like  Tivoli from IBM, EMC Networker, Data Protector and Barrios is in there,  Bacola is there and many others that I forget about.  But okay, it makes it easy. So the the main point of relax recovery is to  make your task for disaster recovery as easy as  possible and design it in that way that it can be  configured in a flexible way that fits your needs.  All right, maybe that sounds maybe simple but okay.  How does it work? Of course your your operating system  consists out of disks so the basic thing is that it has to make  a layout, save the layout of your disk environment, your logical volumes, your  mirror disk, your hardware rates, software rates, whatever.  And all your file systems, how it is labeled, is all written into a  configuration file and that is kept on your life system but also on your  recovery system. And also the bootloaders that you're  using, the major bootloaders that are in use.  Furthermore, then it stores the files. If you request it for it, if you use  say an external commercial backup solution, then it only stores your  basic system layout and when you restore it, it  recreates the disks and then we restore from the backup and that's part of the  integration. And also very important, create the bootloader,  the boot system that you use to recover from it.  And all that can be done and is done online. So there's no  offline thing needed. It's all you can just install it on  predictor system, configure it and it will work. Never use it on  the prediction system without prior testing in a  development environment of course, similar to your production.  Never trust the tool as such. Even I say it's a good tool but  never trust it. Always test it. The recovery,  basically it can, the recovery system can be  made on a nice environment. It can be on a network via pixie booting  or it can be on a new USB disk. That are the three main  recovery possibilities that are in practice today.  Besides the recovery operator system, which is in fact from your live  operator system, it's the same kernel that we're using.  So that's always nice. We copy a lot of other tools that you may  require to do some rescue environment. You can use the rescue  image to do file systems and repairs for example, not  only to recreate everything from scratch. That's not always the  needed but okay. So therefore we copy a lot of tools.  And everything, your complete operating system or rescue  operating system is in RAM. So you never touch your live disk  unless you decide to do it to do a recover.  Of course your live disks will be overwritten.  So this is a graphical overview of the backup flow and output flow.  If you look in the configuration files of relaxant recovery you will see  a backup definition and an output definition.  The output definition is for the rescue image. So where you want your rescue  image to be, do you want it to be an ISO and the location where you want to  store it. And the backup is a fact about your  operating system. If you do a backup  netafacilist for the main default one, that's for the tar or the r-sync  internally. But if you want to, for example, using  a Tivoli backup, then you define backup equals TSM for example,  or DP for data protector. Then you're saying to Rear,  okay, your data of your operating system is in  another backup solution. So we don't use the internal one. And the  internal one is effect tar or r-sync. So that is the main differences. And in  backup you see the internal tar r-sync and the external ones  can be any kind of open source or commercial backup solution.  It also integrates, for example, with duplicity.  This is also a nice cloud backup solution.  The uses of Rear, you have effect to three main parts. You have to make rescue.  That's to make your rescue image only, not backup, only the rescue image.  That's its effect. If you're using an integrative  of a commercial or an open source backup solution, it only  requires a make rescue because you don't need to make backup because  make backup is meaningless at that point. Make backup is  meaningful if you're using an internal backup solution,  like tar, r-sync, whatever. You can also run make backup only. There are many  others. But I would say make backup and the  recover of the two main options within Rear to do  one and make backup is to make a rescue image including the backup.  And the recover is effect if you have a crash and you want to recover  from the backup that was made prior. And as already said, before you can easily  integrate it with, for example, config to HTML with a find quite handy  to have a configuration management overview in text or HTML  included on the ISO, USB, or on the network environment.  So you can always look what was the configuration of my system  before I need to restore. Also very handy if you want to do cloning, for example,  because Rear can be used for cloning systems and it is used for cloning  systems with a lot of comparations.  So you want to get started. These are the main  pointers to the website. Also to the RPMs net solution because we  haven't daily built of Rear with the OpenSuzi built  services. This is a very helpful tool that  OpenSuzi and Susi provides to the developers.  So anything that we, once we do and check in into our hit repository,  then my server will detect it on a nightly basis and will automatically  rebuild and push it to the OpenSuzi build service.  And the next day you have it available or a few minutes later.  On the different architectures for Dabion, Ubuntu,  Fedora, Sless, REL, Sentos, etc. etc.  Furthermore, the official source is on a jitgup,  like I mentioned already. Also the issues are there. If you have an issue with it  you can report it over there. And maybe you know it or you don't know  it. Rear has been added to the base operating system  of Red Hat. So you will find in the 7374 still  the Rear version 172 with I think it's the 9th subversion that they created  with big fixes that were in our issues. But in the 7.4 that has been  released in beginning of August this month, it's rebased to the Rear  version 2.0. And that is a major step from the Rear  1 versions to the 2 because we had it in the re-numbering.  And also a lot of new features and stability was included in Rear.  So if you're still using a Rear 1 version,  it's better to go to the Rear 2 because it's more stable,  it's much more tested also with automated testing,  for example. But okay, this one to say it's in the official  repositories of Red Hat and also Sless has the more of less  official repositories in there thing. One of our main developers is also  working for Sless, I'm for Susan. So that helps of course.  Right, I'm coming to the topic of today in fact,  it's testing Rear. Because Rear is already more than 10 years old  and in the beginning we were using physical hardware,  a lot of virtual machines to test it, but it is very cumbersome because you have  made some modifications, you get new features from  users who want to integrate their backup solution, for example, to give an  example. And we have to blindly trust that it  works. And of course we are all developers and nobody is  perfect. I'm also not perfect of course. We make mistakes  and sometimes mistakes are integrated or checked in automatically without being  tested properly. And it's in fact almost impossible to test  everything because like I said in the beginning  Rear is working mainly built for Intel, but  now the IBM is pushing hard. One of the developers is also now from IBM  for the PPC and PPCLE46 environment. So it's growing fast in that community.  Of course I don't have a lot of, I don't have any PPC  system at home, so it's for me difficult to test on that kind of hardware.  So therefore this environment was also built and IBM is also trying to  integrate Rear and made the testing to test their  new features in it. So it's just too broad to test everything. We do our best  to test the basic stuff. The internal backup solution is tested.  I can test the ISO, I can test the PPC booting. So that's  convenient and also for example for Barrios, if our Barrios uses in here,  that's the open source backup solution, that's the integration they asked me to  to help them to do the integration testing with Barrios.  And that's also done with the automated testing tools.  Like I said, it will be a challenge to test everything, but we try to do our best.  Rear Automated Testing Effect is an open source project, make it very clear,  but it was initially written for customers who paid for it.  Because we're doing more than 20 years open source projects for free  and still have to live as an independent consult. So it's nice to get paid sometimes.  So doing the automated testing is kind of finding a balance between continuous integration  and automated testing. Which is not easy if people here are in DevOps  environment that they probably know. It's always a battle. And even here it's a battle.  But like I already mentioned, the open source build service is a very good first step.  Because that provides me with an automated way of building the RPMs and they are available.  Which is nice. So I don't have to do it myself every day for different platforms.  Which is good. It helps for developers. And we do get feedback from a lot of developers.  It's on operate systems which I do not have or do not use.  They come back with features, enhancements, with bug fixes and also with bugs and forms.  And we do have developers which are game helping us and left.  And we some have a few developers who are staying already on board for several years.  Which is nice. We have now five key developers in our team.  Which is good. Like I said from Suzy, we have a very good one.  From IBM. We have Slomo Shapir myself. And we have also a guy from HPE from Slovenia.  Who is also a very excellent person which is helping us.  We have some previous very good ones. Which are now left the team but still in the background sometimes.  But okay. That's open source. It's coming and going.  And we are always lucky to have some good people on board.  And if they lose their focus and do something else, that's fine.  Know our feelings. We are glad with all the help that we can get.  Support is done via the main channel is Jitschup. The issues.  There is a lot of free support as I said. But you can also have commercial support via my company.  All right. The automated testing. We are coming closer to the demonstration. That's good.  So it was initially written for the customers who have a support contract or a subscription contract.  And currently the first time that I talked about the automated testing was of Fosden.  Beginning this year. And then I only had it for CentOS 7.  And on the LIFERT. So I gave a demonstration with my Linux laptop with LIFERT. KVM LIFERT.  But now we have Ubuntu. And to have it more convenient.  Also integrated into the Oracle VirtualBox. And I will give a demonstration with my Mac.  Just to prove that it works on a non-native Linux system as well. So I have a different  operating systems at home. I have using an own Linux, KVM Linux with Oracle VirtualBox.  But here it will using my Mac system. Just to prove it.  The next slide will show you the overview of the setup. But the first machines that I use  are provisioned automatically with Ansible. In February this year at Fosden it was still  using a script. And also gave a talk at CBIT. And then somebody in the audience  talked to me and said, why don't you do it with Ansible or something else. That's a very good  idea. Because I am now also on DevOps. And it was also a good exercise for myself to dig into  Ansible and to do the complete provision of the virtual machines. That means what do I do  with the operating system is a box. So updating it, installing rear, of course,  all the dependencies of rear will be installed automatically. But for example, if I am using  barriers as an integrated test, I have to install barriers. On the server side I have to configure  it. You have to become a barrier server. And the client has to be configured. So these things are  all automatically done. And it's done for Ubuntu as well. Because they are always between the  different versions, different dependencies. It's a bit awkward, but that's life, of course.  So that's all covered by Ansible. And this demonstrates I will not do the provision from  scratch because that will take 10 minutes. I will just demonstrate the admitted testing itself.  But if there is time, I know later on I can always show you that provisioning is working with Ansible.  So in this setup, what it will do is automatically it will launch a client and a server virtual machine.  It will automatically create a disaster recovery image from the client, make a backup to the server.  If I'm using Oracle virtual machines, virtual box in this case, that means booting needs to  be done from the host system. If I'm using, for example, KVM live-word, I can boot from my server  VM, which is not possible from virtual box. But that's the limitation of virtual box.  So it doesn't mean just a different way of starting up your recovery image. But that means that the  client will, in this case, this demonstrates I will be using the internal network, netFS solution,  and I'm using tar to make a backup of my client system to the server VM. But the same can be  done with Barrios. If time needs, I can always demonstrate it. In this case, I will boot via  Pixi, but can also boot via ISO. When the backup is completely made, I will stop the client  VM and automatically start the recovery VM. The recovery VM will automatically do a restore  from scratch. It will recreate your disk layout, restore everything from scratch.  And that is done. It will reboot and your client VM will be online again. So that's the basically  of automated testing. This is the graphical overview. On the top side, you will see the host system.  And I call it the vagrant host because vagrant is used to drive everything. And on the left side,  you see the client in the middle of the server, and there are the recovery VM. So there are three  VMs created. The client and server are already provisioned. I did it on front. So they are  containing everything. But, okay, that every time that I start, you admit testing system,  I do not reprovision everything for the same operating system. If I want to use CentOS and  it's provisioned, and I want to test a new version of my relaxation recovery version,  I do not to reinstall the operating system from scratch. That's not needed. I just have to update  the relaxation recover version effect and restart the backup. So the reprovisioning is not  required every time. It's only required when I want to switch from CentOS to Ubuntu because that's  kind of logically. You can destroy your box and recreate it automatically. That's done automatically.  So in different ways. So you have an internal network. It's the 33 network, which I'm using to  do the backup internally between the VMs. And you also have an external network,  and the HCP network, which I can talk to the internet. If you want to download the latest  relaxation recovery version from open source build services, it's going to the internet to  download it. And you will see it in the demonstration. All right. And to boot the recover with Pixi,  it boosts from the hypervisor. If I'm using LipFort, it's going that way. So that's the  difference between KVM LipFort and the virtual box situation. This Mac system is using the  virtual box. So I'm booting from the hypervisor. But okay, you don't see the difference effect.  And that's all covered by the script of the automated testing. It's also a script.  Automated testing. It's written in BAS. It's open source.  You can find it. I will show you the pointer later. But to test the environment, I'm using  Vagrant. That's the main feature. Vagrant and virtual box of KVM. If you're using KVM, then  you also need the LipFort plugin. And HIT is also required to download the latest version of the  automated testing software and to clone the Vagrant files, because the Vagrant files are  included in the project. I can show you. I will demonstrate it anyhow. So  to conclude before I dig into the demonstration, it's open source. Everybody can use it, test it  out and give feedback to me or to the project. What is missing, maybe prefixes, whatever.  It's always welcome. New code is written only for customers, in fact. It's not doing it for free.  Unless it's a serious work, of course. You can clone it. This is the address.  You just have to go into the directory and type minus gates for the help. The automated test  is the main script. There's only one script effect. And the rest is sensible. Playbooks.  By default, you can use the Vagrant user to log in. You can also use the root user to log in.  And the password for both is Vagrant. That's also auto provisioned by the Ansible scripts,  playbooks, re-scipes, whatever. As said, there are different ways to log in. You can use the  Vagrant subsystem to log in to your VMs. Or you can use secure shell. Or even you can use VNC.  Or it's a virtual box. You can use the console of the virtual box, of course.  By default, there are also three ports generated. That's for VNC that you can connect to the client,  server, or the recover virtual machine. Right. You want to try it yourself? That's in fact  minus p is the provisioner. If you're using KVM, livevert, then you have to give livevert.  If you're using virtual box, the default is virtual box. Then you don't have to give any  option at all. All right. There is also a testing part. But the testing part which is existing  today is effect and donation from Red Hat using BeakerLip. And these scripts are written in  the BeakerLip language effect. They're using that tool, if I had the Red Hat testing system,  whilst there are main system BeakerLip, to test internally the stability of relax and recover.  And that is still used at Red Hat. But as we are using more and more the automated testing system  for other operating systems, like vendors, like Debian, Ubuntu, Slash. So I'm thinking very hard  to replace these scripts by another system like a bus automated testing system. But that's for later.  I can't demonstrate it. It's working because Red Hat is using it and it's integrated in this tool  with the minus t option. Well, it proves the fact the stability effect of your  OS, whatever you want to test. And now the most important stuff we give in demonstration.  Is it visible? Have to go up a bit, maybe?  All right. Here you can see the main script.  It's over here. The minus h is the help effect. And it shows you different options like the minus d.  It's the distribution. The default is center seven. And that was the first customer who asked for it.  But now there are two others, like the Ubuntu 14 and the Ubuntu 16. You just have to mention it.  If you want to know, okay, how do I know? Just do an ls. And you will see here the Ubuntu,  these are the effect of the operating systems. These are the test scripts. These are the internal  rearscripts. These are the test stuff. And the templates is effect the, it's always interesting  to look at it. These are the configuration files which are used internally for testing  here. You can see this is the ISO booting with netfs and fs. This is the booting with  URL type. This is default. And this is for BIOS. BIOS def names is for the Ubuntu.  And this is with ISO for CentOS. That will change in one of the upcoming rears versions.  I will do auto detection of BIOS def names. But okay, going too deep maybe. That's if you're using  SAP or these kind of hardware, then most of the time you're using BIOS def names. It's not  eaten at zero, but it's the real physical hardware names that you will see. And in this case,  Ubuntu was using the physical hardware and in the recovery it was the aliasing effect. So it  was a bit awkward. I made a different configuration file for it. But okay, it doesn't matter.  Right, for this test, we will be using, let's see, still have time. Yes, we will be using  CentOS 7. So that's the default. Using Pixi, it's also the default. First box is the default.  So I don't have to give any parameter here. I can just announce it. Let's announce it.  So it tells me CentOS 7 will start up Vagrant, the providers, and then second the client.  It tells me there's already an update for the box itself. Okay, that's if I have to do box  inload for that. So that's also useful to replace the box. And if you have a new box,  then I have to do a reprovisioning, of course. But anyhow, it doesn't matter for the moment.  It will start up the client. It will start up the server. Very soon you will see the server coming up.  There is the server. Again, the same kind of text will appearing. So this is pure Vagrant.  Vagrant is driving the startup of your boxes. And it's using internally in the back end,  virtual box. Okay. I always find it nicer to demonstrate something. And then it's easier to  understand the complete picture. So now, yep, the client and server are online. Just five  seconds delay. Just if I want to, was mistaking, control C. If you do it now, it will destroy the  boxes. And if the boxes are destroyed, then you have to do reprovisioning. And that takes more  than 10 minutes. So hands off. Check if the networks are online. That ping test is okay.  That means that the client and server can talk to each other. Of course, if you want to back up.  Here it is. Yep, there's already a new rear uploaded. It goes very fast. Sorry for that.  This is the configuration file that is being used. That is coming from the templates. It's copied  automatically. Every time you start, it's automatically recopied to the client. And it's  already making a backup. And you can see here, it's the first from the 7th of August that is being  used. Relics and recover. It's making a snapshot of your bootloader file systems. And now it's  already making a backup to the client environment. Which is in this case, automatically, because  it's designed to sit that the backup is going to the TF2P area of your virtual hypervisor effect.  You have to do nothing for it. It is built into the tool that it knows which kind of  hypervisor is being used. That takes only a few minutes. Is there something more  interesting to know maybe here? Automatically add the kernel options with the net.if names equals zero.  That is for using the alias names. So to disable the bias def names.  This is convenient because last year I gave a workshop at the open source backup conference.  And we had issues with that. I never had issues with Lippford. But most of the students were using  virtual bugs that problems with the alias names of your interfaces. But if you add this to the  kernel options, then it goes away. So you see backup. Backup URL. Okay, it's done. Backup was  successful, it says. Okay, the backup client area will be on the pixie boot area will be made  readable. I'm halting the client for some machine now. And starting the recovery machine. I'm using  another box. This is an empty box. Again, it's the same vacant. It's driving, driving the complete  stuff. And now this is virtual box taking over. You will see it's pixie booting. So it's really  live booting. No cheating. It will, because it's virtual box, the first network will not work.  But this one, the second one will respond. And then it will load automatically your kernel and  your initial renders, your rescue image from relaxation recover. It will boot it up. You can  see it will reconfigure the network. And it's already starting with reformatting your internal disk.  So your disk layout and automatically the restore of your data or operator system will be restored  automatically. That goes quite quickly. Afterwards, you will also see that it will rebuild the  kernels. Not the kernel, of course, but the initial rend disk. Because it's another size of disk. So  it has to see it's another box. That means internally it's another system. So what I'm doing here is  effect kind of cloning. I'm cloning the client on another fresh hardware. And you can see it here.  For every version of your kernel, it will rebuild your initial rend disk of that operator system,  the client in this case. It's just rebuilt on another system. You will see it at least twice.  And it takes longer effect almost than the restore of your OS. Once that is done, you will notice  that the system will reboot. And, well, then we hope, of course, if the relaxation recover does not  contain any bugs, that it will come up automatically. So here is the rebooting in 30 seconds. That's  also something done by relaxation recover. So that's a timing that I can make shorter or longer,  even zero. But, okay. It's always nice if you see an error that you can interrupt and check your  restored environment. Okay. 30 seconds takes some time. Okay. Stopping automatically the system,  and it will restart. You will see automatically that, again, it will pixie boot. But the pixie  boot configuration was automatically changed that it will boot from the first internal disk,  not from the ISO or network anymore. So that's all done automatically in the background.  And here it is booting up your client system. Takes a few seconds. And, oops, there is the  prompt. You can just log in with vagrant as password. And, well, the client is there.  I recovered the client VM on the recovered VM. I had to do nothing except running this script.  And I can do that every day. I can do it for different backup solutions for backup. For the  moment, it's only barriers. But I can do the same thing for barriers on central seven, Ubuntu 14,  and Ubuntu 16. And whatever is missing can be added, of course, but preferably by sponsoring.  That was my main talk. The slides that will be provided contain a fact and summary of the complete  recovery steps made by the script. And also the pointers to the main site, etc., etc. So,  are there any questions so far? Because this was my talk effect.  No questions? Yes?  What is the effect of maintaining this? Because there are new kernel versions,  not a line, I think. So, how much effort goes into this?  A lot of effort. You can see here, this is the issues.  There are, in general, around 70, 80 issues open all the time. And we are working with,  let's say, three people, not full time, but in our spare times. And also, like the person from  Suzy is almost working 80% of his daytime. And the guy from IBM is working, let's say, 40,  sometimes 50% of his daytime on cases. Because, like I said, in my talk, IBM is hot these days for  the SAPA, SAPANA environment on PPC environments. The customers are requesting it that there is  a decent, relaxed, and recovered solution for it. Disaster recovery solution, I must say.  And therefore, they put a lot of effort to test it on their platform, of course.  I know.  Okay, so the question was, how do we keep up with the rolling new features that are coming from  operating systems from new versus, et cetera, et cetera. Indeed, that's a very intensive labor.  I'm using to do that for 10 years every new time. For example, Ubuntu, from going 14 to 16  major versions, they were split up from ISO Linux into Pixel Linux and what was it,  USB Linux, and stuff like that. All these packets, dependencies were different,  namings, so you have to foresee that. And that is, well, normally, in most cases,  when there is a new version, we ask the community, common users like yourself, are testing it.  And a lot of feedback are coming from guys like you that make issues and say, okay,  doesn't it mint something? And it fails for that and that reason. And then we ask,  so meet the log files because it really has an excellent log basis. And we can find a common  solution for it. That takes time. We do have customers who have a support contract,  which is much faster, of course, because we are paid to do that. I'm working for customers  who are renting me for maintaining their relax and recover basis, which is for 10,000 systems,  making the engineering, that goes better. But okay, it's going so fast, it's not easy to maintain  everything and sometimes it takes months to get something fixed. But the main basis,  let's say, for the main versions of Linux, being less red-head, Ubuntu, Debian, is working fine.  That's okay. Maybe another question, because I mean, I really will drive this out and  I'm really loving that the whole search, you know, that you have backup and recover and I mean,  nobody wants backup. I mean, backup is, everybody wants to recover. Everybody wants to be able to  recover. I mean, don't care about backup. I mean, you need to do it, but you want to be able to  recover. And it's really important that you know that it works. And now, I mean, the closing  subsystem in Windows, getting hopefully better, better, so we have any intention, maybe, I mean,  that is basically really no such thing in the Windows. So is there any chance that this could  someday also backup and recover a Windows system? Oh, the question is, can relax and recover backup  a Windows system? The question is yes. Only on Laptop that contains a Linux system.  It's called backup equals block loan. That's written by one of our colleagues, DeVlat from  HP here from Slovenia. He wrote it and it's working fine. But okay, it's not real, it will never work  natively on Windows, I think. It's a Linux software and it's running on a broad version of Linux.  Different kinds of OS, different kind of hardware. We love to see the ARTS 64 hardware to being  incorporated because we got already questions from Ret yet and asked them, okay, can we have some  sponsoring from you? And it goes quiet. So it's always just a regular between doing it for free  and doing, because if I were for real, I cannot be paid for other projects. But I love to wear  for real. Another question? You said earlier, there is all the interconnection of batch screens.  So why does the hardware come in very often? Should it be built for any kind of  processing? The question is, rear is written in batch. Why is the  differentiation in the different hardware required? Well, the question is very easy.  Different hardware is using different boot technology. The PPC environment is using at  least two different boot technologies, the Yaboot and Lilo. Within the Intel, you have the  ultra legacy BIOS way of booting that can be used, Grip, Grip 2, Lilo, and Grip 8 these days,  or UAFI. That's using a different kind of boot, Grip 2 most of the times. But even within  UAFI, there are different boot technologies. So the way of booting within that area is different.  So all these things are already inside rear, and that is a moving duck effect. So new technology  is added. We are always coming after defects. Well, most of the times, that's okay. It's  only a matter of time that it's get integrated. And that's why there are different in rear,  you will see different tracks for different hardware technologies. Okay. If there are no  more questions, I thank you for your time and for your interesting questions. Have a nice day.  Thank you.
froscon2017-1957-relax-and-recover_automated_testing_0_47.png froscon2017-1957-relax-and-recover_automated_testing_48_167.png froscon2017-1957-relax-and-recover_automated_testing_168_215.png froscon2017-1957-relax-and-recover_automated_testing_216_599.png froscon2017-1957-relax-and-recover_automated_testing_600_659.png froscon2017-1957-relax-and-recover_automated_testing_660_755.png froscon2017-1957-relax-and-recover_automated_testing_756_995.png froscon2017-1957-relax-and-recover_automated_testing_996_1847.png froscon2017-1957-relax-and-recover_automated_testing_1848_2171.png froscon2017-1957-relax-and-recover_automated_testing_2172_4547.png froscon2017-1957-relax-and-recover_automated_testing_4548_4667.png froscon2017-1957-relax-and-recover_automated_testing_4668_6059.png froscon2017-1957-relax-and-recover_automated_testing_6060_7979.png froscon2017-1957-relax-and-recover_automated_testing_7980_10343.png froscon2017-1957-relax-and-recover_automated_testing_10344_14579.png froscon2017-1957-relax-and-recover_automated_testing_14580_16943.png froscon2017-1957-relax-and-recover_automated_testing_16944_18635.png froscon2017-1957-relax-and-recover_automated_testing_18636_20663.png froscon2017-1957-relax-and-recover_automated_testing_20664_23003.png froscon2017-1957-relax-and-recover_automated_testing_23004_23579.png froscon2017-1957-relax-and-recover_automated_testing_23580_26603.png froscon2017-1957-relax-and-recover_automated_testing_26604_29483.png froscon2017-1957-relax-and-recover_automated_testing_29484_29603.png froscon2017-1957-relax-and-recover_automated_testing_29604_29735.png froscon2017-1957-relax-and-recover_automated_testing_29736_33251.png froscon2017-1957-relax-and-recover_automated_testing_33252_37091.png froscon2017-1957-relax-and-recover_automated_testing_37092_39587.png froscon2017-1957-relax-and-recover_automated_testing_39588_42671.png froscon2017-1957-relax-and-recover_automated_testing_42672_43751.png froscon2017-1957-relax-and-recover_automated_testing_43752_45335.png froscon2017-1957-relax-and-recover_automated_testing_45336_46127.png froscon2017-1957-relax-and-recover_automated_testing_46128_46655.png froscon2017-1957-relax-and-recover_automated_testing_46656_48095.png froscon2017-1957-relax-and-recover_automated_testing_48096_48455.png froscon2017-1957-relax-and-recover_automated_testing_48456_48587.png froscon2017-1957-relax-and-recover_automated_testing_48588_48683.png froscon2017-1957-relax-and-recover_automated_testing_48684_48755.png froscon2017-1957-relax-and-recover_automated_testing_48756_50207.png froscon2017-1957-relax-and-recover_automated_testing_50208_52391.png froscon2017-1957-relax-and-recover_automated_testing_52392_52655.png froscon2017-1957-relax-and-recover_automated_testing_52656_57779.png froscon2017-1957-relax-and-recover_automated_testing_57780_58211.png froscon2017-1957-relax-and-recover_automated_testing_58212_58271.png froscon2017-1957-relax-and-recover_automated_testing_58272_61835.png froscon2017-1957-relax-and-recover_automated_testing_61836_63623.png froscon2017-1957-relax-and-recover_automated_testing_63624_64619.png froscon2017-1957-relax-and-recover_automated_testing_64620_64703.png froscon2017-1957-relax-and-recover_automated_testing_64704_64751.png froscon2017-1957-relax-and-recover_automated_testing_64752_64799.png froscon2017-1957-relax-and-recover_automated_testing_64800_64847.png froscon2017-1957-relax-and-recover_automated_testing_64848_69947.png froscon2017-1957-relax-and-recover_automated_testing_69948_70727.png froscon2017-1957-relax-and-recover_automated_testing_70728_70895.png froscon2017-1957-relax-and-recover_automated_testing_70896_72599.png froscon2017-1957-relax-and-recover_automated_testing_72600_73619.png froscon2017-1957-relax-and-recover_automated_testing_73620_73691.png froscon2017-1957-relax-and-recover_automated_testing_73692_74003.png froscon2017-1957-relax-and-recover_automated_testing_74004_74051.png froscon2017-1957-relax-and-recover_automated_testing_74052_74111.png froscon2017-1957-relax-and-recover_automated_testing_74112_74159.png froscon2017-1957-relax-and-recover_automated_testing_74160_74711.png froscon2017-1957-relax-and-recover_automated_testing_74712_74759.png froscon2017-1957-relax-and-recover_automated_testing_74760_74807.png froscon2017-1957-relax-and-recover_automated_testing_74808_74867.png froscon2017-1957-relax-and-recover_automated_testing_74868_74987.png froscon2017-1957-relax-and-recover_automated_testing_74988_75007.png