Paving the transition - Herding unicorns VMs on Kubernetes
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Deutsch, Fabian
After this session you should be able to have an idea of how VMs can be run on a Kubernetes cluster.
 Yeah, so hello, nice to see all of you. Yeah, my name is Fabian Deutsch, and I'm going to speak about Kupert, which is about running virtual machines on containers. I worked on Overt, which is a virtualization solution for a couple of years, and then we went over to work on Kupert, where people from Overt and OpenStack are working on, and I'm working at Red Hat, a Linux company doing Linux stuff. So if we start, then we can say that virtualization is only present today. Oh, the focus of this talk is obviously, or it was the light of this track to say, persistent migration. So how do we get from today to something in the future? And virtualization is only present today, that means it's everywhere. Good morning. So it's getting crowded. I think there's a few things. So virtualization is only present today, that means you have virtual machines everywhere. You've got them, classically, in the data center, where you've got your own virtual machines to run your Oracle database, or you've got them in the cloud, in GCE, in Amazon, in Azure, where you can create gazillions of instances with the snip of a finger. But in this case, it's really about you or about us. How do we get our virtual machines over to the present day? And it's not, I mean, today we just don't have virtual machines anymore, but we also have containers. They came up in the recent years, and I guess that you all know containers. So who of you has used containers before? Nice. And who of you has used virtual machines before? Oh, that's perfect. There you go. So containers are here as well. The question is, if we look at them, and we look at them, and we see them, and we see all the hypes, and we see how much is done with containers, they're so fancy tools with so fancy names, and they look and taste and smell a little bit like virtual machines, but just better because it's new and it's hyped, it can do everything. You won't have any bugs. And then you do get to the point where you ask yourself, how can I replace my VMs with containers? There are several reasons for that. I mean, efficiency is one obvious point. You can say that more containers fit on a host than VMs do. They're quicker to launch and eventually easier to manage because they are text-driven. You have a Docker file, and you can write in there whatever you want, and you just rebuild the image if you want a new version. So that's really handy. In VMs, you need bigger provisioning tools. You can use Ansible, Assault, or Templates and OpenStack. So containers are, and we need to grant that, are really easy to handle. And so at some point, who has asked himself if you can replace your VM with a container? Okay, at least a few. Why didn't you ask yourself why you want to replace them? You want to start from scratch? You want to start from scratch. You want to build everything separate from you and Chinean? Yeah, starting from scratch. Like a new car, you don't want to take your old car and give it your paint. Yeah, so I think we're good on time, so we can make a little extra course. So the thing is, if we speak about replacement, then we need to ask ourselves, can they really substitute each other? So can containers actually replace VMs? But as said, you look at them, they look the same. You can do similar stuff. Oh, the management tools around them are also pretty similar. So you've got Kubernetes and there's Docker's form. And I read one of these new tools from SUSE, what was it called? I don't know, Lighthouse or something like that. So there are a range of tools which help you managing containers and also around the whole life cycle. So there's really a lot of stuff around containers. But then if you look in the detail, what technology is used, what impact does it have for your application? I mean, in the end, we don't run VMs for fun or containers for fun. We run them because we want to deliver an application to users. So we run a VM with a database to give an admin the database he needs or we run a VM to give a user a desktop on demand. So there's a reason why we do it. So why I'm saying that is because we run them and we need to see if we can replace them. So we need to take a look at the technology and see if containers can give us the same as VMs. So for example, if we speak about desktop, which in the university, for example, an admin is providing to users on demand, like for doing a course or a class or programming task or so, they might use a remote desktop solution. That's just not available in containers. So they can do obviously a serial console or so, but it's things you need to be aware of or if you got in a hospital, they'd use remote desktop solutions to have their specific application for handling, I don't know, scans or so remotely sometimes and use smart cards for authentication. So in these classical virtualization world, you've got features to do, pass through of the smart card from your thin terminal to the server, but it's not there yet in the container world. But on the other hand, containers provide different features. Containers are easy to scale. So if you look at Kubernetes, you've got the possibility to take one container and scale it up to hundreds, which is not so easy with VMs on the other side. So there are technical differences, even if they look and feel the same and if they really fit into your use case, that really depends on the application you're supporting for your users. So if you can replace a VM with a container, it really depends on the use case you have. Okay, so in case you can replace a container with your VM with a container, you're all good. So it might be a Java application you can put into container and that's really simple. You don't have to do anything. Well, a year later, you know that you have to do some stuff to really make it efficient and fit your application to container. So it's not easy. But I want to focus more on the case where you can say, no, I cannot replace my VM with a container because of certain constraints. I mean, today, the virtualization world is there for, I don't know, 20 years or so in different stages and the applications we have evolved and they grew in that system. So you have big monolith lifts which are optimized to run in a VM. They make assumptions about the operating system, about how you deploy stuff. So everything we as admins or we as developers did in the last years was built on these assumptions we have for virtual machines. Like, a lot of tools have UIs. So there is the expectation that you have a screen to display that UI and you don't meet that requirement in the container world where you just have, well, you can say it takes UIs, UIs as well, I'm speaking about really GUI applications. And surely you can take your application and change it to meet the container technology but it takes time to change an application. If we think about complex applications, I don't know, for big business processes, you need to change them. You need to adopt them. And so you might say, I cannot replace my VM right now with a container but I want to do that in the future when my application is ready for that. Good. So now regardless if you want to start now or if you want to start in the future, at least we are here because we say we see the transition is happening. We see we are in the VM world today and we want to get started with containers in production really, I mean, seriously, we want to get started. So the question is, do we start from scratch? We've shiny new things and everything works. I think if you look at the containers themselves, it absolutely makes sense. I mean, you want to build up your container infrastructure from scratch and look at it. But if you look at the application, are you using virtual machines production? Okay. So would you say the application you provide to your customers or to your users would start that application from scratch? Or you would rewrite it to fit into the container world? No. Yeah. That's quite a lot of work. So I think the infrastructure will be started from scratch to host containers but the application yourself, that is not something you want to rewrite. So we need to see how can we, I mean, we do, we provide our application to customers because we make money and hopefully. And we don't want to rewrite that just for the reason of moving to new infrastructure. So in reality, we need to find a way how we can gracefully move over. We are here. That is a fact. And we need to provide and continue to provide our product to customers or to users. And the question is, how can we still do that? How can we keep providing that application to users even if we move our infrastructure, for example, in the direction of containers because of the benefits they have? And they support different development models. I mean, agile is a big word in that world and DevOps and oh, there are so many cool words. So it's not that we want to write our application from scratch but we rather want to see that we support an evolution so that we take our application and gracefully move it into the direction to be able to run containers. So we want to evolve it. So how can we do that? And that is slowly the point where I want to get at with this presentation is how can we support the transition from the classic data center and Cloud World today with the classic application with the legacy? You can call it legacy applications we have today. And into that new world where you have decomposed applications, microservices, running on service measures and you can find another or more buzzwords here as well. So the classical stack, very simplistic, is that you have your infrastructure, your bare metal infrastructure, so you've got your storage, ZAN or NUS, dedicated, you've got your network switches and you've got, obviously, you've got your physical hosts and they are aggregated into some kind of management plane. So you've got your virtual machine management application, VMware, Overt, OpenStack. There are many others, just a few. And on top of them, you've got your virtual machines running, which you need to keep running. And I'm continuing to highlight that because we, I mean, I'm paid by Red Hat and we live in the reality that we need to see how we can really support our customers and that is really what is driving us. How can we get them from today into the future? We need to give them an enablement and we want to support them in that. So that is where we are today. And then we say, okay, let's start from scratch and bring up our new management plane, which is optimized for containers. So we need storage again. We need network again. We need service again. And then we'll have a management plane dedicated to containers. And you see that on the right side with the blue box. But that means you have twice the infrastructure. Depending on what your setup is, you need to dedicate your storage device to this container use case and you need to dedicate a network, an overlay network, whatever, to that container use case. So I mean, if you want to start, if you want to POC something, if you want to get a feeling for it, that's totally fine. But we speak about really the series migration after that. You see containers work for me and somehow I need some time to migrate my stuff, but that works then. In that case, you probably don't want to maintain two production-grade infrastructure parts in your company. Actually before, you actually want to maintain a single one. You want to backup a single Xan or NUS and you want to make sure that the single instance of switches are running and that you don't have to double everything. Actually, there's another setup what you could do. You could say, I take my existing virtual machine setup and on top of that, I build my container management plane. So everything above the orange box is then virtualized. But the problem here is that you still have two infrastructure layers because you've got the real one at the bottom and the middle one where you need to provide virtual storage. So you need to have a VM which is serving storage. Networking is not so problematic, but you still need to manage it in the virtual machines. So you still have an overhead because you have got two different layers where you need to provide infrastructure either for virtual machines or for containers. So it's not, I think it's good for the POC case and people do that very often. Who did that? Who used virtual machines to play with containers? I am very good. That's good for the statistics. So yeah, it's very convenient. I agree, but again, in practice, it's then you still have to maintain two infrastructures. And that's where we get to Kuber. So Kuber tries to exactly go into that area. So Kuber provides you the ability to run containers and virtual machines on the same infrastructure. So you don't have to provide two different, two distinct set of components for storage network and just physical host, all the stuff you need. Eventually two data centers or two rooms physically depends really on what you want to provide to your customers. So how does it look? It's simple. So Kuber allows you to use the same management plane and to use the same physical storage and network, virtual storage and network and other components you have in your setup. The management plane in this case is Kubernetes, but why is it better or why is it good in our opinion? It's because it allows you. The orange box is so large because we have the assumption that today you have many virtual machines and you start to play with containers. But over time, you want to move stuff over where it makes sense. So we said some applications can be moved faster than others depending on their requirements. Some might not be able to be moved ever if we think about security aspects. If somebody in the room is working with security, you might know that you need VMs to have real complete OS isolation. I don't want to go into the container details, but containers have these, yeah, they effectively shared the kernel as you all know and VMs provide the ability to not share the kernel, but have your own kernel per VM, which is good if you want to do, I don't know. Some testing on that kernel, oh, for, oh, yeah, testing is a good example. If you want different architectures on a single host, then you still want to have VMs because containers is not the component which can give you that. So you can slowly move in this slide. You've moved more and more stuff from the virtual machine side to the container side. And the good thing is the storage network is not twice there in the picture. In reality, that really helps admins to keep the oversight over their components. Yeah, and in the end, it doesn't look much different. As said, you might want to keep some VMs where it doesn't make sense to move to containers. Whenever you've got questions, interrupt me. I think we've got enough time. Now that we've looked at these overview, why we want to have Q-Bird because we look at it from the reality standpoint, you cannot build everything from scratch again because it's all money, it's all time in between invest. So we wanted to see how we can provide a migration path for your applications. So let's take a look, a little bit, a close look at how this looks technically. So this was pretty high level so far and I want to get a little bit more into detail. So Kubernetes is the management thing we built upon, which is classically the framework to run containers on a cluster, on a number of hosts. But it doesn't provide the ability to run virtual machines. There are different projects. I mean, we have Q-Bird, Q-Bird is the very best here by far. But we've got other projects like Wirtled and Rancher and they try something similar. So they also run virtual machines on Kubernetes but with a different focus. I'll get to that in a minute. So in very simple pictures, if you now add Q-Bird to the picture, then you're able to run virtual machines. All right. So that's actually a slide missing. I should have added that. So why is Q-Bird different than, for example, Wirtled or Rancher? Oh, who has heard of Wirtled or Rancher or Clear Containers? Rancher. Rancher. Great. Ah, really, really interesting to see. I think Rancher is the project which is closest to Q-Bird, by the way. But the difference to Rancher, because it seems to be common sense here, is no, not a difference to Rancher, but in general, our take is that we want to express VMs as they are. So if you look at VMs and containers or if you look into one of the Kubernetes introductions, then if you look at the difference between VMs and pods, you see that they have different properties. And we spoke about that. VMs can have architectures. Pods don't have that. It's implicitly dictated by the architecture below. The VMs have kind of, because on multiple displays, pods don't have that. Graphical display at all is not something what pods have. Multiple networks, physical different devices, if you want to test device drivers or do device pass through, that is not something you can simulate in a pod. You can for sure provide a network interface to a pod, but it's not device, it's a network device, but it's not driven by a real driver. It's not, you know, you do, now I've got into those details, which are difficult to explain. So you don't really have device drivers for that. For example, if we look at the VTHs and the kernel, they are not physical devices. They are concepts inside the kernel to connect containers to the host namespace. I'm saying that because we think that there are enough differences between pods and virtual machines that VMs need their own definition. So if an number is taken, and please correct me, it has been some time, at least Wirtlet is one of the approaches which is running VMs and Kubernetes, but they implicitly derive the VM from the pod specification. So there, for example, they have some work around, they use annotations to specify the number of CPUs, but if you want to reuse a specific disk, or they use the volumes of pods to attach disks, but setting displays, adding USB devices, that's not possible because they derive it from the pod spec. And clear containers is doing the same. So they look at the pod and derive a VM from the pod spec. But as said, the pod spec does not provide enough details to define any VM you can have. If we look at third operating systems from Microsoft, like Windows, then it's a fact that you need to have a stable hardware ABI over time. So your device should not change, otherwise you're asked to read register or reauthenticate. I don't actually know the name of the process, but you need to call Microsoft to get your, oh, activate, to activate your Windows again if the hardware devices change. So we would really like to keep these VM API or ABI stable over time with the Rancher or VertLit approach. Yeah, even with Rancher. With the Rancher, VertLit and Clear Container projects are possible. If you do updates and Qmer changes below, the ABI will change. Depending how often that happens, you need to activate it at some point. So we said we want to expose the VM ABI, so the virtual hardware, completely as an entity in Kubernetes. And that's an important, that is the difference between QBert and other solutions. So if you look at the VM ABI in QBert, then you will see, we can actually do that, that you can express really the VM details. And that is necessary to keep that VM ABI stable over time. Who's familiar with LibBert, eventually? That's good. So LibBert is such a component which has been around in the virtualization area of Linux for a long time. And we use it internally. And that is why the VM definition looks a little bit similar to the Domic's ML if you have looked at LibBert. And here you also see, and that's actually the first bullet on that slide is, oh, the second bullet, that we have a new resource type for VMs. That is what you just saw. So we have a specific dedicated VM type where you can specify many details. But it will take some time to recover all the details because there are problems in the details. Oh, if you think about performance, a very good example. VMs often tune for performance. And then you get to funny things like Numa, where you need to make sure that certain virtual processes are tied to the right physical processors. And if you do a device path through that device is really aligned with the device node of the virtual CPU. And that is stuff Kubernetes does not care about yet. They are working on it. But we need the type control over it because we speak about series VMs. We really want to provide production-ready VMs which you can use and support over time. And so what we can do is we can use this extensive VM API I've just shown to pin CPUs to certain physical CPUs. Right. So I spoke about the API. One thing is that we want, and I'll go to the first bullet, is that we provide everything in pods. And why do we do that? If you look at virtual adventurer-clear containers, take off three of them. Currently, you cannot run it on any Kubernetes cluster because it makes assumptions about the hosts below. So you need to install, in the virtual case, it assumes that QMOS around, it assumes that the relevant kernel modules are around. So that is stuff which, if we look at Kubernetes, it's all about the cluster. You usually don't care about what operating system is below Kubernetes because you can run Kubernetes on Debian, Linux, on Fedora, on Suzy, CoreOS, whatever. And you usually don't care what is below. I have my container. I want to run it on the cluster. So the new platform is the cluster, not the host anymore, which is good. But in these existing solutions, what we identified as problems, they do rely on the underlying host, and that limits where you can run the solution. So we try to provide all the dependencies in pods so that you can deploy, like any application in Kubernetes. You don't have to provide anything else. Actually we try to be that good that we can act as a Kubernetes add-on so that you've got namespace awareness and that you integrate with the networking concepts that you recognize the network policies and all that kind of stuff. We try to be a good citizen as well by being based on an operator pattern. That's a pattern introduced by CoreOS. The important part here is that we try to behave so even if we are virtualization and virtualization conflicts with containers in certain fields, but we still try to behave nicely on the cluster. So we also want to be declarative, like you can declare pods on the state of pods, and we follow this pattern by implementing the operator pattern. So the last bullet point here is VMs live inside pods. On the one side it's an implementation detail, but often it does people who care about it, so that's why I named it. So VMs are not pods in our implementation, but VMs live in the resource group of a pod. What's the difference and why does it matter anyway? So we have the ability to say that we run, and it might come at some point, but it's not so important yet. So if we run pods, the only unit Kubernetes is aware of in the case of a specific workload is a pod, all other concepts are built on pods, so our high level concepts to pods. And what we said is all right, so Kubernetes tracks pods. So we want to see, especially tracks resource usage and it applies limits to pods. So what we needed to make sure is that we really, that on the one hand Kubernetes is able to track VMs the same way. So that is why we place them in pods, but you could still say, well, but why don't you, for example, implement VMs in Kubernetes? And that is where we get to these conflicts again. If we implemented VMs really inside Kubernetes, then we have the problem that certain functionality is just not there in Kubernetes because it doesn't fit with those concepts. For example, if we think about live migration, who used VM live migration in practice? A few people. It's very convenient because you don't have to bring down and call your customers and say, yeah, it will be back in 20 minutes, but you migrate your VM and it often works. We had a lot of time to optimize that process, but it's not there in the container world. You can implement that. I wonder how good it will become, but it's conceptually not something you have with containers. It's a stateless, how was it called yesterday? Immortal. So they can go away and you just re-insentiate a new one. That is not something you want to do with production VMs of legacy applications. You want to keep them around. That is what we live migrate them. So we want to have live migration, but we knew that we cannot bring it into Kubernetes because it clashes with the concept of containers. So that is why we say that whole VM logic is a top Kubernetes. It's a layer of top Kubernetes and it's leveraging the concept of Kubernetes. And that's why we use pods. We see a pod as a harbor for a VM. So you bring a pod and that's where we place the VM. And if we do a migration, we just create a new pod, then we move the VM from one pod to the other. And Kubernetes can kill it afterwards. So the state of the VM is free not tied to a pod, which is good because that is how Kubernetes handles pods in containers. But we can still deliver our functionality of doing the live migration because we just see them as resources, constraints, or resource containers for VMs. If I'm getting too technical or too detailed, please let me know. Or if you've got questions, also let me know. All right. So we looked at the VM API. That's good. And because we're getting already to the end, I think we'll schedule for 45 minutes. We can get our hands on. I hope it works. I'm confident it does. So has anybody tried Minicube before? All that's cool. For all the people who didn't lift the hand, I would really recommend it. Minicube is so nice to try. If you want to get your hands off with Kubernetes, give Minicube a try. It's really like, I'll just Google Minicube or Bing Minicube or Duck Duck Go Minicube. You will find it. So the demo I'm going to show, maybe we can still switch again, is based on Minicube. I'm using the VMDriver KVM, which is just important because it's Fedora and it's Linux. And you can have different drivers. Important is to use the network plug-in CNI. It can actually clone that demo, and I hope it works for you as well. This slide will be public later on in case you really want to go to try. So what I'm doing now is that we will be quite some output. So what the demo does, it clones, cupid from GitHub, so we are all public. It's nothing secret even if we do it as a company. Or so it's all public. We have all the issues there. You're welcome to contribute, welcome to try, welcome to file issues, welcome to fix them. So what happened is we cloned that repository and used the manifest, so we provide all kinds of manifest. So here we see we've got a VIRT controller and we've got a VM resource. So the VM resource is our own VM definition, the API we saw before, the controller is related to the operator pattern. So it's watching for new VM definitions and then acting upon them. We've got the handler, which is the demon set and we're deciding on every node and making sure that if the VM is getting scheduled to a specific host that we really launch a VM if it's getting scheduled to a specific host. And all these components were now deployed on MeCube. If you do it for the first time, it can take depending on your internet connection sometime because images are pulled down and it's still early phase, so the images are quite large. It will get better over time. So if we look what kind of pods are running now, that's too large. Yeah, that's better. We see that we've got a few components running. So we've got HA proxy. If anybody wonders why HA proxy, then lift your hand, but we can ignore it. We've got an iSCALT demo target because we, I'll get to that in a minute, we've got Libre, Spice Proxy to give the graphical console access from outside the cluster, VIRT API, the VIRT controller. I explained that the launcher test VM is actually, that is the pod which I said, which is launched to provide the resource container for the VM. And we've got the VIRT manifest, which we want to get rid of, or maybe not, let's see. So these are really components and it's a stock MeCube deployment. So what can we do now? We looked at the pods and what we can do, oh yeah, we can take a look at the Libre container, for example, to just confirm that we really have a VM running. And that's really nice if it works. So here, see, so what I did is, so we've got the VIRT launcher test VM. And that was scheduled on a specific host. But this pod for opening the resource group for that VM was launched, the handler saw it and told Libre to spawn a VM based on the VM API, which is also declared. Actually, let's take a look at the VM API. Oh yeah, oh, that's nice. Because we've got our own type in Kubernetes, we can actually use the whole CubeCTL tools to retrieve these commands. And actually, the goal of CubeRid is we want to be friendly citizens and we want to work with Kubernetes. So you know today, CubeCTL is pretty much tuned to what the Kubernetes case is. So it has all the verbs to work with containers, but it doesn't support live migration, for example. So we want to work with the Kubernetes guys. We are actually providing our input there to make sure that, for example, in this CubeCTL case, we can add our own verbs. And if you're tracking the Kubernetes progress, and you might have seen that Kubernetes or CubeCTL now, I think 17 or 18, is supporting binary plugins on the hosts. So we now have the ability to provide our add-on to CubeCTL to then provide verbs which are relevant to virtualization. So we can have our verb k, CubeCTL, live migrate, and the VM name, for example. So here, see, that is really the VM definition as it was shown on the slides. And that is used to bring up the VM on the host. And if you look above, it has the feeling of Kubernetes, even if you can express all the VM requirements. You can work with labels, you can work with taints and tolerations. So you can really leverage the infrastructure. A word on the infrastructure. So we started with that on the first slide. We said containers and virtual machines, if you want to run them today in production, you need separate infrastructure basements for both. In CubeCTL, it's not really used in this demo. I actually should update that. If we look here, we see that this little section says that we add a disk to the VM, but it's not using a persistent volume from Kubernetes in this case, but rather directly in iSCSI device or iSCSI target. But what we work on, and it's actually merged, so I should update this, that you can use regular persistent volumes of Kubernetes. Instead of for pods, you can use them for VMs. There are certain constraints. So virtual machines don't support file systems at disks. You need a raw block device or a device which is providing raw block device like semantics. So we work with Kubernetes. We actually now have in Kubernetes 1.8 the first raw block device support, and we try to engage them. We try to really get it into Kubernetes on the long run that we have stable support over time. And you can use PVs with the VM specs, so instead of referencing an iSCSI device, you can use a PV like you could use it for pods. And the semantics about exclusive or the access modes, multiple reader, multiple writer are recognized. What we're currently working on because the second large infrastructure part is networking. And that's actually more tricky than storage. And we're also working here or collaborating with different parts to see that we get networking for VMs aligned with the infrastructure of Kubernetes. What does it mean? We just want to connect in a simple way VMs to the same networks as Kubernetes is using. It's more complicated than it sounds at the beginning. All right, so the demo is running. We showed the VMs. We showed that it was running. If you follow the complete demo, then you can download the binary tool, and then you can do stuff like... What's the keyword demo? Yeah, and then keyword cluster, Qubectl. Then you can do stuff like console test VM. The issue is that we lately moved to a new binary tool, which is a pre-stage to having that at OnFurQubectl, which I did not install yet. And we can then directly access the console. You can even say Qubectl spice test VM to then open a spice window. Yeah. All right. Then we can move back to the slides. And that's quick. But yes, it was the last part. The demo is often good to put at the end because you never know if it works. So yeah, that's it from my side. Thank you very much so far. And the questions. Very good, yeah. Yeah, you said network with problem. I could explain why. Because what I understand is usually there's some sort of C9 glitch on your host, and you can do a potential VM to be closed. Yeah. So the question was, what is the problem with networking? And the problem with networking is conceptual. So many problems with Kubernetes and VMs are conceptual. So what you get in a pod is not an interface, but that's implicitly what a pod is getting. And the core concept is that every pod has an IP. So Kubernetes is dictating that the pod is getting an IP. But in the virtualization world, you don't give an IP to the VM. I mean, you might do it if you have a DHCP server on your LAN, but you effectively give it an interface. So layer 2 connectivity. And so we need to say how we can. So we've got, actually a colleague of mine is implementing now layer 3 connectivity for VMs so that we can really play nice with stock Kubernetes networking. But the core problem is we need to address is we want to give VMs the ability, like in our classical data center today, to choose their own IP and just provide them the layer 2 networking where they live on. Yeah, you could say, we could actually say, we don't want to do that. We just want to keep it the Kubernetes way and manage the IPs for them. So do I peer-dress management? But we know that it takes time to legacy application support of living with the fact that they now get an IP and that they cannot dictate it anymore. So but our first step is playing nice and we're close to get there and that will work with CNI. So we are requesting new interfaces from CNI for each Nick, for each virtual Nick we assigned to a VM that's currently working progress. And I'm happy to show it next year or at KubeCon. Any more questions to that or? OK. The man in blue behind you. Do you think it's really worth the effort to move all the VMs to a new platform for having in mind that it's going to be dead in the US? Oh. You have all the old infrastructure like the Ember and Liburn and you have the administrators of the regular with all the tools and everything that you really want to offer new infrastructure. Where you already know it's going to be dead. So the question was why do we invest anyway if VMs are going to die? That's a good question. We should probably stop now. I think the assumption is that VMs don't die so quickly and we just even if it's quite some effort. After all, it's not so much. It's actually pretty nice. It's a technical challenge so it really is fun working on that. Thank you. We've got five minutes left. But we need, I think it's worth the effort because it really takes time and sometimes it takes more, we all know it often takes more time than you estimated at the beginning. So we know it will take time. And if we look at how many, you know, a lot of companies still get money from moving stuff from UNIX to Linux and UNIX is pretty old. So we know that transitions take time and we want to provide a solution for this transition path. But we don't want to have you being stuck with the infrastructure. I think companies will eventually stop supporting old infrastructure. I don't know if VM where now really goes up and just supporting containers. What do you do with vSphere? vSphere is not seeing any updates anymore. That's what we say, you still need to run your VMs but you also need to be up to date to have security updates and that kind of stuff. So we say it's worth the effort to say we allow you to move the VMs to the new infrastructure but keep your classic VMs running until you are ready. And eventually, and there are reasons often again where you see that VMs are needed. Like for relief, if you want to do testing on certain architectures, CI is such a big thing and it works on the application level with containers but on the lower levels it just doesn't. And that is what we know that VMs will stick around. Also for certain isolation techniques, we know that the isolation of containers because of the single kernel approach, some say it's not as good, some say it's sufficient. But regardless of that discussion, there are regulations which require to have a strong virtual machine isolation. Eventually these regulations will see an update as well but so far they haven't. In these cases, to meet regulations of governmental regulations, you need to run VMs. So, and the questions often if really VMs will die or not, to me. There are head packs who isolate desktop applications, I think we are sentencing, I'm wondering what would be the, by the way, running desktop applications as a container instead of a cloud pack. Oh, a cloud pack as a container, Ranta. Yeah, so what was the question? Can't. Flat back is, who knows flat back? Oh, if you've got too much time, take a look in flat pack, we really need to give it some power, give it some love. So flat back is a user friendly or for users of container format, no not a container format. It's a container runtime targeted for unprivileged users. So you don't need root access to run containers. And the question was, why can't we use classical containers to run applications? Why should we use flat back? And I think it's, after all, it's the same. Technically, they also use C groups, they also use namespaces. They use username spaces, that's why they can run unprivileged, which is an important characteristic. But image wise, they are very similar. They might use different techniques on the low level, but the core concept, it's a container. They ship dependencies in a container, so to me, it's not so much different. Or was there something I missed in the question? I understood that we would, there needs to be an answer to one, there's an application in certain situations, that because you couldn't use containers in one way or another, flat back would be something that would solve. Okay. So the question was, and I did, got it wrong, sorry. The question is, why can't we use flat back to run desktop applications? It's difficult to run flat back, Windows applications in flat back if Wine doesn't support it. And it's difficult to run a PowerPC or Power8, Power9 application in flat back if you've got an xx864 host. And that is where VMs again come into the game. So we said that initially in the talk, why, what are the differences between VMs and containers? And it's like, there are certain, it really depends on the application. If it's like T-Markz, I would say, all right, you don't need a VM for T-Markz, you can use flat back. If you run calc or a cal, the calendar application on the terminal, you can use flat back. But if you use, oh, even if you use GNOME, if you use GNOME calculator, you can use flat back. It's, you know, it's, you can get it for the architecture you want, and it's Linux based. That's no problem. But if you really get to the details, you want to have, you only have a graphical application which requires, oh, well, let me, it depends on the requirements of your application. If you need remote support, what kind of strong isolation you need, if you need a different kernel because of different architecture or different features, those, those, those questions explain to the decision. Actually, we had a diagram early on where it said it depends. You can port some applications directly to containers, but often legacy applications which have grown over years, you cannot directly move them containers. You need a transitioning period to adopt the applications. I hope I got it better now. Okay. Sure. Anything else? Who of you will now try Q-Bird? That's good. Any more hands? I would like to see more. No, very good. If you've got any questions, get us on IRC. Oh, I missed to put IRC there. Twitter, GitHub, Filo, contact us. We're all friendly. We're all humans. So, FTC, you're there. And thank you very much for your time. Good morning. Thank you. Thank you. Thank you. Thank you.
 Yeah, so hello, nice to see all of you.  Yeah, my name is Fabian Deutsch, and I'm going to speak about Kupert, which is about running  virtual machines on containers.  I worked on Overt, which is a virtualization solution for a couple of years, and then we  went over to work on Kupert, where people from Overt and OpenStack are working on,  and I'm working at Red Hat, a Linux company doing Linux stuff.  So if we start, then we can say that virtualization is only present today.  Oh, the focus of this talk is obviously, or it was the light of this track to say, persistent  migration.  So how do we get from today to something in the future?  And virtualization is only present today, that means it's everywhere.  Good morning.  So it's getting crowded.  I think there's a few things.  So virtualization is only present today, that means you have virtual machines everywhere.  You've got them, classically, in the data center, where you've got your own virtual machines  to run your Oracle database, or you've got them in the cloud, in GCE, in Amazon, in Azure,  where you can create gazillions of instances with the snip of a finger.  But in this case, it's really about you or about us.  How do we get our virtual machines over to the present day?  And it's not, I mean, today we just don't have virtual machines anymore, but we also  have containers.  They came up in the recent years, and I guess that you all know containers.  So who of you has used containers before?  Nice.  And who of you has used virtual machines before?  Oh, that's perfect.  There you go.  So containers are here as well.  The question is, if we look at them, and we look at them, and we see them, and we see  all the hypes, and we see how much is done with containers, they're so fancy tools with  so fancy names, and they look and taste and smell a little bit like virtual machines,  but just better because it's new and it's hyped, it can do everything.  You won't have any bugs.  And then you do get to the point where you ask yourself, how can I replace my VMs with  containers?  There are several reasons for that.  I mean, efficiency is one obvious point.  You can say that more containers fit on a host than VMs do.  They're quicker to launch and eventually easier to manage because they are text-driven.  You have a Docker file, and you can write in there whatever you want, and you just rebuild  the image if you want a new version.  So that's really handy.  In VMs, you need bigger provisioning tools.  You can use Ansible, Assault, or Templates and OpenStack.  So containers are, and we need to grant that, are really easy to handle.  And so at some point, who has asked himself if you can replace your VM with a container?  Okay, at least a few.  Why didn't you ask yourself why you want to replace them?  You want to start from scratch?  You want to start from scratch.  You want to build everything separate from you and Chinean?  Yeah, starting from scratch.  Like a new car, you don't want to take your old car and give it your paint.  Yeah, so I think we're good on time, so we can make a little extra course.  So the thing is, if we speak about replacement, then we need to ask ourselves, can they really  substitute each other?  So can containers actually replace VMs?  But as said, you look at them, they look the same.  You can do similar stuff.  Oh, the management tools around them are also pretty similar.  So you've got Kubernetes and there's Docker's form.  And I read one of these new tools from SUSE, what was it called?  I don't know, Lighthouse or something like that.  So there are a range of tools which help you managing containers and also around the whole  life cycle.  So there's really a lot of stuff around containers.  But then if you look in the detail, what technology is used, what impact does it have for your  application?  I mean, in the end, we don't run VMs for fun or containers for fun.  We run them because we want to deliver an application to users.  So we run a VM with a database to give an admin the database he needs or we run a VM  to give a user a desktop on demand.  So there's a reason why we do it.  So why I'm saying that is because we run them and we need to see if we can replace them.  So we need to take a look at the technology and see if containers can give us the same  as VMs.  So for example, if we speak about desktop, which in the university, for example, an admin  is providing to users on demand, like for doing a course or a class or programming task  or so, they might use a remote desktop solution.  That's just not available in containers.  So they can do obviously a serial console or so, but it's things you need to be aware  of or if you got in a hospital, they'd use remote desktop solutions to have their specific  application for handling, I don't know, scans or so remotely sometimes and use smart cards  for authentication.  So in these classical virtualization world, you've got features to do, pass through of  the smart card from your thin terminal to the server, but it's not there yet in the  container world.  But on the other hand, containers provide different features.  Containers are easy to scale.  So if you look at Kubernetes, you've got the possibility to take one container and scale  it up to hundreds, which is not so easy with VMs on the other side.  So there are technical differences, even if they look and feel the same and if they really  fit into your use case, that really depends on the application you're supporting for your  users.  So if you can replace a VM with a container, it really depends on the use case you have.  Okay, so in case you can replace a container with your VM with a container, you're all  good.  So it might be a Java application you can put into container and that's really simple.  You don't have to do anything.  Well, a year later, you know that you have to do some stuff to really make it efficient  and fit your application to container.  So it's not easy.  But I want to focus more on the case where you can say, no, I cannot replace my VM with  a container because of certain constraints.  I mean, today, the virtualization world is there for, I don't know, 20 years or so in  different stages and the applications we have evolved and they grew in that system.  So you have big monolith lifts which are optimized to run in a VM.  They make assumptions about the operating system, about how you deploy stuff.  So everything we as admins or we as developers did in the last years was built on these assumptions  we have for virtual machines.  Like, a lot of tools have UIs.  So there is the expectation that you have a screen to display that UI and you don't  meet that requirement in the container world where you just have, well, you can say it  takes UIs, UIs as well, I'm speaking about really GUI applications.  And surely you can take your application and change it to meet the container technology  but it takes time to change an application.  If we think about complex applications, I don't know, for big business processes, you  need to change them.  You need to adopt them.  And so you might say, I cannot replace my VM right now with a container but I want to  do that in the future when my application is ready for that.  Good.  So now regardless if you want to start now or if you want to start in the future, at  least we are here because we say we see the transition is happening.  We see we are in the VM world today and we want to get started with containers in production  really, I mean, seriously, we want to get started.  So the question is, do we start from scratch?  We've shiny new things and everything works.  I think if you look at the containers themselves, it absolutely makes sense.  I mean, you want to build up your container infrastructure from scratch and look at it.  But if you look at the application, are you using virtual machines production?  Okay.  So would you say the application you provide to your customers or to your users would start  that application from scratch?  Or you would rewrite it to fit into the container world?  No.  Yeah.  That's quite a lot of work.  So I think the infrastructure will be started from scratch to host containers but the application  yourself, that is not something you want to rewrite.  So we need to see how can we, I mean, we do, we provide our application to customers because  we make money and hopefully.  And we don't want to rewrite that just for the reason of moving to new infrastructure.  So in reality, we need to find a way how we can gracefully move over.  We are here.  That is a fact.  And we need to provide and continue to provide our product to customers or to users.  And the question is, how can we still do that?  How can we keep providing that application to users even if we move our infrastructure,  for example, in the direction of containers because of the benefits they have?  And they support different development models.  I mean, agile is a big word in that world and DevOps and oh, there are so many cool  words.  So it's not that we want to write our application from scratch but we rather want to see that  we support an evolution so that we take our application and gracefully move it into the  direction to be able to run containers.  So we want to evolve it.  So how can we do that?  And that is slowly the point where I want to get at with this presentation is how can  we support the transition from the classic data center and Cloud World today with the  classic application with the legacy?  You can call it legacy applications we have today.  And into that new world where you have decomposed applications, microservices, running on service  measures and you can find another or more buzzwords here as well.  So the classical stack, very simplistic, is that you have your infrastructure, your bare  metal infrastructure, so you've got your storage, ZAN or NUS, dedicated, you've got your network  switches and you've got, obviously, you've got your physical hosts and they are aggregated  into some kind of management plane.  So you've got your virtual machine management application, VMware, Overt, OpenStack.  There are many others, just a few.  And on top of them, you've got your virtual machines running, which you need to keep running.  And I'm continuing to highlight that because we, I mean, I'm paid by Red Hat and we live  in the reality that we need to see how we can really support our customers and that  is really what is driving us.  How can we get them from today into the future?  We need to give them an enablement and we want to support them in that.  So that is where we are today.  And then we say, okay, let's start from scratch and bring up our new management plane, which  is optimized for containers.  So we need storage again.  We need network again.  We need service again.  And then we'll have a management plane dedicated to containers.  And you see that on the right side with the blue box.  But that means you have twice the infrastructure.  Depending on what your setup is, you need to dedicate your storage device to this container  use case and you need to dedicate a network, an overlay network, whatever, to that container  use case.  So I mean, if you want to start, if you want to POC something, if you want to get a feeling  for it, that's totally fine.  But we speak about really the series migration after that.  You see containers work for me and somehow I need some time to migrate my stuff, but  that works then.  In that case, you probably don't want to maintain two production-grade infrastructure parts  in your company.  Actually before, you actually want to maintain a single one.  You want to backup a single Xan or NUS and you want to make sure that the single instance  of switches are running and that you don't have to double everything.  Actually, there's another setup what you could do.  You could say, I take my existing virtual machine setup and on top of that, I build  my container management plane.  So everything above the orange box is then virtualized.  But the problem here is that you still have two infrastructure layers because you've got  the real one at the bottom and the middle one where you need to provide virtual storage.  So you need to have a VM which is serving storage.  Networking is not so problematic, but you still need to manage it in the virtual machines.  So you still have an overhead because you have got two different layers where you need  to provide infrastructure either for virtual machines or for containers.  So it's not, I think it's good for the POC case and people do that very often.  Who did that?  Who used virtual machines to play with containers?  I am very good.  That's good for the statistics.  So yeah, it's very convenient.  I agree, but again, in practice, it's then you still have to maintain two infrastructures.  And that's where we get to Kuber.  So Kuber tries to exactly go into that area.  So Kuber provides you the ability to run containers and virtual machines on the same infrastructure.  So you don't have to provide two different, two distinct set of components for storage  network and just physical host, all the stuff you need.  Eventually two data centers or two rooms physically depends really on what you want to provide  to your customers.  So how does it look?  It's simple.  So Kuber allows you to use the same management plane and to use the same physical storage  and network, virtual storage and network and other components you have in your setup.  The management plane in this case is Kubernetes, but why is it better or why is it good in  our opinion?  It's because it allows you.  The orange box is so large because we have the assumption that today you have many virtual  machines and you start to play with containers.  But over time, you want to move stuff over where it makes sense.  So we said some applications can be moved faster than others depending on their requirements.  Some might not be able to be moved ever if we think about security aspects.  If somebody in the room is working with security, you might know that you need VMs to have real  complete OS isolation.  I don't want to go into the container details, but containers have these, yeah, they effectively  shared the kernel as you all know and VMs provide the ability to not share the kernel,  but have your own kernel per VM, which is good if you want to do, I don't know.  Some testing on that kernel, oh, for, oh, yeah, testing is a good example.  If you want different architectures on a single host, then you still want to have VMs because  containers is not the component which can give you that.  So you can slowly move in this slide.  You've moved more and more stuff from the virtual machine side to the container side.  And the good thing is the storage network is not twice there in the picture.  In reality, that really helps admins to keep the oversight over their components.  Yeah, and in the end, it doesn't look much different.  As said, you might want to keep some VMs where it doesn't make sense to move to containers.  Whenever you've got questions, interrupt me.  I think we've got enough time.  Now that we've looked at these overview, why we want to have Q-Bird because we look at  it from the reality standpoint, you cannot build everything from scratch again because  it's all money, it's all time in between invest.  So we wanted to see how we can provide a migration path for your applications.  So let's take a look, a little bit, a close look at how this looks technically.  So this was pretty high level so far and I want to get a little bit more into detail.  So Kubernetes is the management thing we built upon, which is classically the framework to  run containers on a cluster, on a number of hosts.  But it doesn't provide the ability to run virtual machines.  There are different projects.  I mean, we have Q-Bird, Q-Bird is the very best here by far.  But we've got other projects like Wirtled and Rancher and they try something similar.  So they also run virtual machines on Kubernetes but with a different focus.  I'll get to that in a minute.  So in very simple pictures, if you now add Q-Bird to the picture, then you're able to  run virtual machines.  All right.  So that's actually a slide missing.  I should have added that.  So why is Q-Bird different than, for example, Wirtled or Rancher?  Oh, who has heard of Wirtled or Rancher or Clear Containers?  Rancher.  Rancher.  Great.  Ah, really, really interesting to see.  I think Rancher is the project which is closest to Q-Bird, by the way.  But the difference to Rancher, because it seems to be common sense here, is no, not  a difference to Rancher, but in general, our take is that we want to express VMs as  they are.  So if you look at VMs and containers or if you look into one of the Kubernetes introductions,  then if you look at the difference between VMs and pods, you see that they have different  properties.  And we spoke about that.  VMs can have architectures.  Pods don't have that.  It's implicitly dictated by the architecture below.  The VMs have kind of, because on multiple displays, pods don't have that.  Graphical display at all is not something what pods have.  Multiple networks, physical different devices, if you want to test device drivers or do device  pass through, that is not something you can simulate in a pod.  You can for sure provide a network interface to a pod, but it's not device, it's a network  device, but it's not driven by a real driver.  It's not, you know, you do, now I've got into those details, which are difficult to  explain.  So you don't really have device drivers for that.  For example, if we look at the VTHs and the kernel, they are not physical devices.  They are concepts inside the kernel to connect containers to the host namespace.  I'm saying that because we think that there are enough differences between pods and virtual  machines that VMs need their own definition.  So if an number is taken, and please correct me, it has been some time, at least Wirtlet  is one of the approaches which is running VMs and Kubernetes, but they implicitly derive  the VM from the pod specification.  So there, for example, they have some work around, they use annotations to specify the  number of CPUs, but if you want to reuse a specific disk, or they use the volumes of  pods to attach disks, but setting displays, adding USB devices, that's not possible because  they derive it from the pod spec.  And clear containers is doing the same.  So they look at the pod and derive a VM from the pod spec.  But as said, the pod spec does not provide enough details to define any VM you can have.  If we look at third operating systems from Microsoft, like Windows, then it's a fact  that you need to have a stable hardware ABI over time.  So your device should not change, otherwise you're asked to read register or reauthenticate.  I don't actually know the name of the process, but you need to call Microsoft to get your,  oh, activate, to activate your Windows again if the hardware devices change.  So we would really like to keep these VM API or ABI stable over time with the Rancher  or VertLit approach.  Yeah, even with Rancher.  With the Rancher, VertLit and Clear Container projects are possible.  If you do updates and Qmer changes below, the ABI will change.  Depending how often that happens, you need to activate it at some point.  So we said we want to expose the VM ABI, so the virtual hardware, completely as an entity  in Kubernetes.  And that's an important, that is the difference between QBert and other solutions.  So if you look at the VM ABI in QBert, then you will see, we can actually do that, that  you can express really the VM details.  And that is necessary to keep that VM ABI stable over time.  Who's familiar with LibBert, eventually?  That's good.  So LibBert is such a component which has been around in the virtualization area of Linux  for a long time.  And we use it internally.  And that is why the VM definition looks a little bit similar to the Domic's ML if you  have looked at LibBert.  And here you also see, and that's actually the first bullet on that slide is, oh, the  second bullet, that we have a new resource type for VMs.  That is what you just saw.  So we have a specific dedicated VM type where you can specify many details.  But it will take some time to recover all the details because there are problems in  the details.  Oh, if you think about performance, a very good example.  VMs often tune for performance.  And then you get to funny things like Numa, where you need to make sure that certain  virtual processes are tied to the right physical processors.  And if you do a device path through that device is really aligned with the device node of  the virtual CPU.  And that is stuff Kubernetes does not care about yet.  They are working on it.  But we need the type control over it because we speak about series VMs.  We really want to provide production-ready VMs which you can use and support over time.  And so what we can do is we can use this extensive VM API I've just shown to pin CPUs to certain  physical CPUs.  Right.  So I spoke about the API.  One thing is that we want, and I'll go to the first bullet, is that we provide everything  in pods.  And why do we do that?  If you look at virtual adventurer-clear containers, take off three of them.  Currently, you cannot run it on any Kubernetes cluster because it makes assumptions about  the hosts below.  So you need to install, in the virtual case, it assumes that QMOS around, it assumes that  the relevant kernel modules are around.  So that is stuff which, if we look at Kubernetes, it's all about the cluster.  You usually don't care about what operating system is below Kubernetes because you can  run Kubernetes on Debian, Linux, on Fedora, on Suzy, CoreOS, whatever.  And you usually don't care what is below.  I have my container.  I want to run it on the cluster.  So the new platform is the cluster, not the host anymore, which is good.  But in these existing solutions, what we identified as problems, they do rely on the underlying  host, and that limits where you can run the solution.  So we try to provide all the dependencies in pods so that you can deploy, like any application  in Kubernetes.  You don't have to provide anything else.  Actually we try to be that good that we can act as a Kubernetes add-on so that you've  got namespace awareness and that you integrate with the networking concepts that you recognize  the network policies and all that kind of stuff.  We try to be a good citizen as well by being based on an operator pattern.  That's a pattern introduced by CoreOS.  The important part here is that we try to behave so even if we are virtualization and  virtualization conflicts with containers in certain fields, but we still try to behave  nicely on the cluster.  So we also want to be declarative, like you can declare pods on the state of pods, and  we follow this pattern by implementing the operator pattern.  So the last bullet point here is VMs live inside pods.  On the one side it's an implementation detail, but often it does people who care about it,  so that's why I named it.  So VMs are not pods in our implementation, but VMs live in the resource group of a pod.  What's the difference and why does it matter anyway?  So we have the ability to say that we run, and it might come at some point, but it's  not so important yet.  So if we run pods, the only unit Kubernetes is aware of in the case of a specific workload  is a pod, all other concepts are built on pods, so our high level concepts to pods.  And what we said is all right, so Kubernetes tracks pods.  So we want to see, especially tracks resource usage and it applies limits to pods.  So what we needed to make sure is that we really, that on the one hand Kubernetes is  able to track VMs the same way.  So that is why we place them in pods, but you could still say, well, but why don't you,  for example, implement VMs in Kubernetes?  And that is where we get to these conflicts again.  If we implemented VMs really inside Kubernetes, then we have the problem that certain functionality  is just not there in Kubernetes because it doesn't fit with those concepts.  For example, if we think about live migration, who used VM live migration in practice?  A few people.  It's very convenient because you don't have to bring down and call your customers and  say, yeah, it will be back in 20 minutes, but you migrate your VM and it often works.  We had a lot of time to optimize that process, but it's not there in the container world.  You can implement that.  I wonder how good it will become, but it's conceptually not something you have with containers.  It's a stateless, how was it called yesterday?  Immortal.  So they can go away and you just re-insentiate a new one.  That is not something you want to do with production VMs of legacy applications.  You want to keep them around.  That is what we live migrate them.  So we want to have live migration, but we knew that we cannot bring it into Kubernetes  because it clashes with the concept of containers.  So that is why we say that whole VM logic is a top Kubernetes.  It's a layer of top Kubernetes and it's leveraging the concept of Kubernetes.  And that's why we use pods.  We see a pod as a harbor for a VM.  So you bring a pod and that's where we place the VM.  And if we do a migration, we just create a new pod, then we move the VM from one pod  to the other.  And Kubernetes can kill it afterwards.  So the state of the VM is free not tied to a pod, which is good because that is how  Kubernetes handles pods in containers.  But we can still deliver our functionality of doing the live migration because we just  see them as resources, constraints, or resource containers for VMs.  If I'm getting too technical or too detailed, please let me know.  Or if you've got questions, also let me know.  All right.  So we looked at the VM API.  That's good.  And because we're getting already to the end, I think we'll schedule for 45 minutes.  We can get our hands on.  I hope it works.  I'm confident it does.  So has anybody tried Minicube before?  All that's cool.  For all the people who didn't lift the hand, I would really recommend it.  Minicube is so nice to try.  If you want to get your hands off with Kubernetes, give Minicube a try.  It's really like, I'll just Google Minicube or Bing Minicube or Duck Duck Go Minicube.  You will find it.  So the demo I'm going to show, maybe we can still switch again, is based on Minicube.  I'm using the VMDriver KVM, which is just important because it's Fedora and it's Linux.  And you can have different drivers.  Important is to use the network plug-in CNI.  It can actually clone that demo, and I hope it works for you as well.  This slide will be public later on in case you really want to go to try.  So what I'm doing now is that we will be quite some output.  So what the demo does, it clones, cupid from GitHub, so we are all public.  It's nothing secret even if we do it as a company.  Or so it's all public.  We have all the issues there.  You're welcome to contribute, welcome to try, welcome to file issues, welcome to fix them.  So what happened is we cloned that repository and used the manifest, so we provide all kinds  of manifest.  So here we see we've got a VIRT controller and we've got a VM resource.  So the VM resource is our own VM definition, the API we saw before, the controller is related  to the operator pattern.  So it's watching for new VM definitions and then acting upon them.  We've got the handler, which is the demon set and we're deciding on every node and making  sure that if the VM is getting scheduled to a specific host that we really launch a VM  if it's getting scheduled to a specific host.  And all these components were now deployed on MeCube.  If you do it for the first time, it can take depending on your internet connection sometime  because images are pulled down and it's still early phase, so the images are quite large.  It will get better over time.  So if we look what kind of pods are running now, that's too large.  Yeah, that's better.  We see that we've got a few components running.  So we've got HA proxy.  If anybody wonders why HA proxy, then lift your hand, but we can ignore it.  We've got an iSCALT demo target because we, I'll get to that in a minute, we've got  Libre, Spice Proxy to give the graphical console access from outside the cluster, VIRT API,  the VIRT controller.  I explained that the launcher test VM is actually, that is the pod which I said, which is launched  to provide the resource container for the VM.  And we've got the VIRT manifest, which we want to get rid of, or maybe not, let's see.  So these are really components and it's a stock MeCube deployment.  So what can we do now?  We looked at the pods and what we can do, oh yeah, we can take a look at the Libre container,  for example, to just confirm that we really have a VM running.  And that's really nice if it works.  So here, see, so what I did is, so we've got the VIRT launcher test VM.  And that was scheduled on a specific host.  But this pod for opening the resource group for that VM was launched, the handler saw it  and told Libre to spawn a VM based on the VM API, which is also declared.  Actually, let's take a look at the VM API.  Oh yeah, oh, that's nice.  Because we've got our own type in Kubernetes, we can actually use the whole CubeCTL tools  to retrieve these commands.  And actually, the goal of CubeRid is we want to be friendly citizens and we want to work  with Kubernetes.  So you know today, CubeCTL is pretty much tuned to what the Kubernetes case is.  So it has all the verbs to work with containers, but it doesn't support live migration, for  example.  So we want to work with the Kubernetes guys.  We are actually providing our input there to make sure that, for example, in this CubeCTL  case, we can add our own verbs.  And if you're tracking the Kubernetes progress, and you might have seen that Kubernetes or  CubeCTL now, I think 17 or 18, is supporting binary plugins on the hosts.  So we now have the ability to provide our add-on to CubeCTL to then provide verbs which  are relevant to virtualization.  So we can have our verb k, CubeCTL, live migrate, and the VM name, for example.  So here, see, that is really the VM definition as it was shown on the slides.  And that is used to bring up the VM on the host.  And if you look above, it has the feeling of Kubernetes, even if you can express all  the VM requirements.  You can work with labels, you can work with taints and tolerations.  So you can really leverage the infrastructure.  A word on the infrastructure.  So we started with that on the first slide.  We said containers and virtual machines, if you want to run them today in production,  you need separate infrastructure basements for both.  In CubeCTL, it's not really used in this demo.  I actually should update that.  If we look here, we see that this little section says that we add a disk to the VM, but it's  not using a persistent volume from Kubernetes in this case, but rather directly in iSCSI  device or iSCSI target.  But what we work on, and it's actually merged, so I should update this, that you can use  regular persistent volumes of Kubernetes.  Instead of for pods, you can use them for VMs.  There are certain constraints.  So virtual machines don't support file systems at disks.  You need a raw block device or a device which is providing raw block device like semantics.  So we work with Kubernetes.  We actually now have in Kubernetes 1.8 the first raw block device support, and we try  to engage them.  We try to really get it into Kubernetes on the long run that we have stable support over  time.  And you can use PVs with the VM specs, so instead of referencing an iSCSI device, you  can use a PV like you could use it for pods.  And the semantics about exclusive or the access modes, multiple reader, multiple writer are  recognized.  What we're currently working on because the second large infrastructure part is networking.  And that's actually more tricky than storage.  And we're also working here or collaborating with different parts to see that we get networking  for VMs aligned with the infrastructure of Kubernetes.  What does it mean?  We just want to connect in a simple way VMs to the same networks as Kubernetes is using.  It's more complicated than it sounds at the beginning.  All right, so the demo is running.  We showed the VMs.  We showed that it was running.  If you follow the complete demo, then you can download the binary tool, and then you  can do stuff like...  What's the keyword demo?  Yeah, and then keyword cluster, Qubectl.  Then you can do stuff like console test VM.  The issue is that we lately moved to a new binary tool, which is a pre-stage to having  that at OnFurQubectl, which I did not install yet.  And we can then directly access the console.  You can even say Qubectl spice test VM to then open a spice window.  Yeah.  All right.  Then we can move back to the slides.  And that's quick.  But yes, it was the last part.  The demo is often good to put at the end because you never know if it works.  So yeah, that's it from my side.  Thank you very much so far.  And the questions.  Very good, yeah.  Yeah, you said network with problem.  I could explain why.  Because what I understand is usually there's some sort of C9 glitch on your host, and you  can do a potential VM to be closed.  Yeah.  So the question was, what is the problem with networking?  And the problem with networking is conceptual.  So many problems with Kubernetes and VMs are conceptual.  So what you get in a pod is not an interface, but that's implicitly what a pod is getting.  And the core concept is that every pod has an IP.  So Kubernetes is dictating that the pod is getting an IP.  But in the virtualization world, you don't give an IP to the VM.  I mean, you might do it if you have a DHCP server on your LAN, but you effectively give  it an interface.  So layer 2 connectivity.  And so we need to say how we can.  So we've got, actually a colleague of mine is implementing now layer 3 connectivity for  VMs so that we can really play nice with stock Kubernetes networking.  But the core problem is we need to address is we want to give VMs the ability, like in  our classical data center today, to choose their own IP and just provide them the layer  2 networking where they live on.  Yeah, you could say, we could actually say, we don't want to do that.  We just want to keep it the Kubernetes way and manage the IPs for them.  So do I peer-dress management?  But we know that it takes time to legacy application support of living with the fact that they  now get an IP and that they cannot dictate it anymore.  So but our first step is playing nice and we're close to get there and that will work  with CNI.  So we are requesting new interfaces from CNI for each Nick, for each virtual Nick we  assigned to a VM that's currently working progress.  And I'm happy to show it next year or at KubeCon.  Any more questions to that or?  OK.  The man in blue behind you.  Do you think it's really worth the effort to move all the VMs to a new platform for  having in mind that it's going to be dead in the US?  Oh.  You have all the old infrastructure like the Ember and Liburn and you have the administrators  of the regular with all the tools and everything that you really want to offer new infrastructure.  Where you already know it's going to be dead.  So the question was why do we invest anyway if VMs are going to die?  That's a good question.  We should probably stop now.  I think the assumption is that VMs don't die so quickly and we just even if it's quite  some effort.  After all, it's not so much.  It's actually pretty nice.  It's a technical challenge so it really is fun working on that.  Thank you.  We've got five minutes left.  But we need, I think it's worth the effort because it really takes time and sometimes  it takes more, we all know it often takes more time than you estimated at the beginning.  So we know it will take time.  And if we look at how many, you know, a lot of companies still get money from moving stuff  from UNIX to Linux and UNIX is pretty old.  So we know that transitions take time and we want to provide a solution for this transition  path.  But we don't want to have you being stuck with the infrastructure.  I think companies will eventually stop supporting old infrastructure.  I don't know if VM where now really goes up and just supporting containers.  What do you do with vSphere?  vSphere is not seeing any updates anymore.  That's what we say, you still need to run your VMs but you also need to be up to date  to have security updates and that kind of stuff.  So we say it's worth the effort to say we allow you to move the VMs to the new infrastructure  but keep your classic VMs running until you are ready.  And eventually, and there are reasons often again where you see that VMs are needed.  Like for relief, if you want to do testing on certain architectures, CI is such a big  thing and it works on the application level with containers but on the lower levels it  just doesn't.  And that is what we know that VMs will stick around.  Also for certain isolation techniques, we know that the isolation of containers because  of the single kernel approach, some say it's not as good, some say it's sufficient.  But regardless of that discussion, there are regulations which require to have a strong  virtual machine isolation.  Eventually these regulations will see an update as well but so far they haven't.  In these cases, to meet regulations of governmental regulations, you need to run VMs.  So, and the questions often if really VMs will die or not, to me.  There are head packs who isolate desktop applications, I think we are sentencing, I'm wondering what would be the,  by the way, running desktop applications as a container instead of a cloud pack.  Oh, a cloud pack as a container, Ranta.  Yeah, so what was the question?  Can't.  Flat back is, who knows flat back?  Oh, if you've got too much time, take a look in flat pack, we really need to give it some power,  give it some love.  So flat back is a user friendly or for users of container format, no not a container format.  It's a container runtime targeted for unprivileged users.  So you don't need root access to run containers.  And the question was, why can't we use classical containers to run applications?  Why should we use flat back?  And I think it's, after all, it's the same.  Technically, they also use C groups, they also use namespaces.  They use username spaces, that's why they can run unprivileged, which is an important characteristic.  But image wise, they are very similar.  They might use different techniques on the low level, but the core concept, it's a container.  They ship dependencies in a container, so to me, it's not so much different.  Or was there something I missed in the question?  I understood that we would, there needs to be an answer to one, there's an application in certain situations,  that because you couldn't use containers in one way or another,  flat back would be something that would solve.  Okay.  So the question was, and I did, got it wrong, sorry.  The question is, why can't we use flat back to run desktop applications?  It's difficult to run flat back, Windows applications in flat back if Wine doesn't support it.  And it's difficult to run a PowerPC or Power8, Power9 application in flat back if you've got an xx864 host.  And that is where VMs again come into the game.  So we said that initially in the talk, why, what are the differences between VMs and containers?  And it's like, there are certain, it really depends on the application.  If it's like T-Markz, I would say, all right, you don't need a VM for T-Markz, you can use flat back.  If you run calc or a cal, the calendar application on the terminal, you can use flat back.  But if you use, oh, even if you use GNOME, if you use GNOME calculator, you can use flat back.  It's, you know, it's, you can get it for the architecture you want, and it's Linux based.  That's no problem.  But if you really get to the details, you want to have, you only have a graphical application which requires, oh, well, let me,  it depends on the requirements of your application.  If you need remote support, what kind of strong isolation you need,  if you need a different kernel because of different architecture or different features, those, those, those questions  explain to the decision.  Actually, we had a diagram early on where it said it depends.  You can port some applications directly to containers, but often legacy applications which have grown over years,  you cannot directly move them containers.  You need a transitioning period to adopt the applications.  I hope I got it better now.  Okay.  Sure.  Anything else?  Who of you will now try Q-Bird?  That's good.  Any more hands?  I would like to see more.  No, very good.  If you've got any questions, get us on IRC.  Oh, I missed to put IRC there.  Twitter, GitHub, Filo, contact us.  We're all friendly.  We're all humans.  So, FTC, you're there.  And thank you very much for your time.  Good morning.  Thank you.  Thank you.  Thank you.  Thank you.
froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_0_59.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_60_455.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_456_515.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_516_695.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_696_995.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_996_2471.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_2472_3239.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_3240_3947.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_3948_5747.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_5748_9707.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_9708_10127.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_10128_12467.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_12468_16331.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_16332_17471.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_17472_17531.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_17532_17579.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_17580_19499.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_19500_21227.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_21228_22559.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_22560_25235.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_25236_32003.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_32004_32327.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_32328_32663.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_32664_32915.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_32916_34703.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_34704_43259.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_43260_43307.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_43308_43415.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_43416_43619.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_43620_44255.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_44256_44555.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_44556_45023.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_45024_45143.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_45144_45239.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_45240_45299.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_45300_46991.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_46992_51263.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_51264_52235.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_52236_52403.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_52404_54755.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_54756_56207.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_56208_58607.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_58608_58655.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_58656_60239.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_60240_60959.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_60960_61643.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_61644_64271.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_64272_64547.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_64548_64715.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_64716_68963.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_68964_69227.png froscon2017-2023-paving_the_transition_-_herding_stroke_unicorns_stroke_vms_on_kubernetes_69228_69475.png