State of the rkt container runtime
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
López Galeiras, Iago
rkt is a modern container runtime, built for security, efficiency, and composability. It is one of the container runtimes supported by Kubernetes but the current implementation (“rktnetes”) doesn’t support the Container Runtime Interface (CRI). The work-in-progress CRI implementation is called rktlet. This presentation will give an update on the rkt project, what new features were implemented recently and what’s coming up. It will also give an update on the state of the rktlet: what features are missing and what workarounds should be removed before it becomes a complete implementation of the CRI.
 Yeah, so in this talk I'm going to talk about Rocket, what's the state of Rocket, where we, what's being new in Rocket, and what's the future of Rocket. And we'll also talk about Rocketlit, which is the part that integrates Rocket with Kubernetes. So first, something's about me. So hi, I'm Iago. I live in Berlin, and I'm one of the Kimphol co-founders. You probably know Kimphol by now because we are sort of organizing this conference. So, but for those of you who don't know us, we are basically a for hire Linux engineering team that focus on core cloud infrastructure technologies. So yeah, one of those cloud infrastructure technologies is Rocket, and we've been working a lot on that in the past. And yeah, we've also retaken this work recently. And yeah, I'm currently a maintainer of Rocket and the Rocketlit. And yeah, there's my GitHub and my Twitter account if you want to find out more about me. So the plan for today is I'm going to give a brief explanation of Rocket. So what is Rocket? And then I will explain how it works. So some of the Rocket internals. I'll do the same for the Rocketlit. What it is, how does it work? I'll explain what's new and yeah, what are the new features we've been working on recently, and then I will explain what's missing. So yeah, we have a vision for the future. So Rocket very briefly is a modern, secure, composable, container runtime. Modern as in we try to take advantage in the latest technologies in Linux specifically like username spaces or overlay effects or whatnot. And come secure as in we try to be secure by default. So all the security options are enabled by default. If you want to disable it, you have to do it explicitly. So you're conscious of what you're doing. And composable because we try to play nice with the rest of the system. We have a good integration with systemd. It's not required, but yeah, we have a very good integration with systemd. And yeah, composable. And ultimately it's just an implementation of the app-c spec, specifically the parts of image format and execution environment. Yeah, so app-c is what we started when we started Rocket to specify the image format and the security environment. And now it's kind of not maintained, but Rocket still uses it heavily. So I will give a brief look at what app-c is. So yeah, it's a standard application container specification. It's an open specification and it defines some associated tooling. Yeah, there's some repos under the app-c organization like AC server or Docker2ACI. CNI was part of the app-c spec, but now it's moved to container networking since it's being used by a broader audience. But for our purposes, we're going to talk a bit about the spec. So it defines four things, image format, which is what an application consists of. So it's basically a table with some manifest associated that says the results image you want, executable you want to execute. Then there's the image discovery part, which is how we translate an image name to an actual URL where the artifact of this image is. And for that, we take advantage of the DNS system. So we have a translation from the name to a DNS namespace. Then it defines spots, which is how applications are grouped and run. And then it defines the executor, which is what the application is going to see when it's executed. So yeah, Rocket is a simple CLI tool. So we don't have any demons that are mandatory. It's written in Golang and it's basically Linux only. And it is self-contained as in we don't require any weird dependencies, just your typical dynamic libraries that you have installing all systems. And it's in its system and district agnostic, although as I said, we optimize heavily for system D. So yeah, being a simple CLI tool, it means that the applications run directly under the spawning process. So there's some diagram there. You can see that there's the spawning process, which can be bash or system D or unit, you name it. Then under that, there's Rocket. And then under that, there's the pod and the two apps in this example. So you can see these stage 0, stage 1, stage 2. So the execution of Rocket is divided in three stages. So basically, stage 0 is the CLI tool. So it takes care of discovering fetching images and rendering them on the disk. Stage 1 is basically the actual container. So that sets up the exact environment for pods. And it manages the processes lifecycle. And it applies resource constraints to. And stage 2 is the app itself. So whatever you're running, NGINX or any app. So one cool thing about stage 1 is that it's swappable. So that means there's different implementations of stage 1, which is pretty cool. So the full one uses Linux namespaces and secrets for isolation. So what you call a Linux container. And it's based on system D and spawn and system D. So we basically run a system D and spawn container. And inside, we run a system D instance that manages all the processes in the pod. There is also a KVM implementation, which uses hardware virtualization for isolation instead of containers. So that's useful when you want a bit more security. And it's based on QMU nowadays and also system D inside the pod. So the part that manages processes is shared between those stages. Then there's also fly, which is basically a no-isolation stage 1. It's just a chisage route. And its purpose is to take advantage of rockets, image handling, and signing, and all that stuff. There's some more, but these are the main ones. So I'm going to stop talking about rockets. And I'm going to introduce Kubernetes for those who don't know it. Yeah, so basically Kubernetes is a container orchestration system. So it gives a developer an API endpoint where it can define which containers the developer wants to run. And the developer doesn't have to care about where they run or how this is done. It just says, I want to run five instances of this container. And then Kubernetes takes care of putting those containers in actual hosts. So you can see there that there's this funny icon on the top. That's a CD. And that's basically a distributed key value system that offers the cluster like a full view of a cluster that's consistent between among different hosts. So that's a distributed primitive that's needed to implement Kubernetes. And those worker kubelet boxes are basically hosts. So on each host, a kubelet is running. And the kubelet is the one that is responsible for creating the actual containers inside the host. So yeah, then the kubelet basically nowadays implements an interface called the CRI. Well, not implements, but defines an interface called CRI. And CRI defines an interface where there are different methods that express the things that you want to do with containers. So for example, the kubelet can say Rampot Sandbox. So a new sandbox will be started in a host. And the kubelet can say, OK, now create this container in that sandbox and then start it or stop it or get the status or all those stuff. So this is implemented via gRPC, which is a remote procedure called by Google. So kubelet will call one of those methods. And there will be something on the other side listening and actually doing the actual operation. So these things are called shims. And yeah, there are several of them. There's CRI Continuity, which is recently been developed a lot by Google. There's Cryo, which I think it reached version 1.0 in the last week or two weeks. And then there's Rocketlit, which is a rocket implementation of the CRI. And yeah, it's bold because that's what I'm going to talk about. So how does it work? Yeah, so you have the kubelet that connects through gRPC, as I said before, to the Rocketlit. And basically what the Rocketlit does when the kubelet requests a new sandbox is called SystemD-Run, and it just starts a SystemD unit that runs Rocket, the Rocket Sandbox. And then the Rocketlit communicates with Rocket just running Rocket commands. And it can ask Rocket to add containers or remove containers or whatnot. So in order to implement this, we had to change the design of Rocket a bit because it was basically a immutable design. So if you start a pod, it will be this way, and that's it until you stop it and you start a new one. So we had to do some changes. These changes are what we call the app experiment. So yeah, as I said before, the CRI operates in sandbox and container terms, not just in pods. So Rocket had to be redesigned, and we added a new sub command for that, which is the sub command app. So you will have a Rocket sub command that matches each one of the methods of this CRI interface I was talking about before. So with this, SystemD was pretty helpful for us because this maps pretty neatly with what SystemD does. So for example, Rocket app add will just add a new unit file to the pod and load it. And Rocket app start will just call SystemCTL start basically. And yeah, this is experimental because it's a pretty big change to Rocket. So for now, it's under an environment variable called Rocket experiment app. So you have to specify that before calling Rocket to use these features because if not, they will be hidden. Another thing that we had to deal with is the logs. So being that we run SystemD inside the pod, the natural way to handle logs is to just use the journal. And that's what we were doing. Rocket was logging to JournalD. The problem is that the CRI interface wants logs in a custom text format. It doesn't want the journal format. So the initial solution, kind of hackish way to solve this, was to have basically another container on each pod that translates the journal logs to the CRI format. So that had some problems because it was, first of all, very resource consuming. So you don't want to have a sidecord container. You just want to have the containers you want to run. And it was kind of unreliable. So sometimes the logs were not there. And you will get weird error messages, which nobody likes. Moreover, we wanted to implement things like attach to the standard input and standard output of a container. And JournalD doesn't have anything to, so we can do that. So the proper solution is what we call the attach experiment. So it's another experiment. And that uses a component called IoTTYmax. And that allows us to write directly to the CRI file format. So this is a simplified version of how it works. So this is the rocket system at the rocket process tree. So you can see systemD and spawn, and then the actual container, which runs systemD inside. So basically every application will connect its standard input, standard output, and error to this IoTTYmax daemon. And then that will pass through the standard input and output to systemD, to an external process, or into a log file. So in this case, only standard output and error. So this is the simplified version of the CRI format. It's just very simple, just to date what stream it is, and then the message. So yeah, this allows us to attach and to write logs in the format directly without having a sidecar like we had before. So basically those were the two things that were the most important things that we had to change for Rocket to work well with Kubernetes. Yeah, I'm going to talk about the recent developments, like in the last two, three months in Rocket. So first we added a feature that a client requested to share the host IPC namespace, because they wanted to do some operations on the IPC namespace. And you need to share it with the host so other apps can actually do that. So Michael Lee Alban implemented that. And then we need to do a lot of fixes to make RocketLit possible. Yeah, a lot of bug fixes, especially with the experiments. Yeah, experimental, so they weren't really working well. And we need to do some integration work between the attach experiment and Kubernetes, because it needs some to put the logs in a specific path, and we need to add some configuration options to do that. And another thing that we did is we switched the default KVM flavor from LKVM, which was not very reliable for us, to QM. So yeah, now it works much better. So yeah, what comes next in Rocket? One thing that I think Casey started but didn't finish is updating CNI to version 06, because we're using 03, and there's a lot of new features in newer CNI versions that we need to update. And we need to stabilize the Rocket experiments and remove all these environment variables. So we need to do a lot more tests and remove those so they can be used out of a box. Yeah, we experimented a bit with using Run C to set up the stage two runtime. So what we were doing until now was using system.dunit files, and those provide a lot of primitives to do a lot of resource limiting, or mainly resource limiting. So Run C gives a bit more flexibility though, and we started looking at using Run C instead of, yeah, I think Casey over there was starting to work. Yeah, so that would be pretty cool. And we did some experiments with CAsync, which for those of you who don't know that, it's a new project by Leonard Pertöding, and I'll talk a bit more about that in the next slide, and also general bug fixing, because there's always bugs, right? So yeah, the CAsync tests are explained here in this graph. So basically CAsync is a mixture between the R-Sync algorithm and a content addressable store, and that allows you to distribute images, for example, in a much more efficient way, because it will only distribute the parts that are missing. And so we wanted to test that with container images and see what the improvements were. So you can see that the blue line is basically just downloading everything. And the red line is what CAsync transfers on the network. So you can see that there are some spikes, and that's when a lot of things change in the image. So in that case, you have to re-send everything basically. But overall, you can see an improvement. And so that's very cool. There's the repo there, where we did the experiments, where you can find scripts to reproduce them. So yeah, we'll probably look a bit more into that in the future. Yeah. So now about Rocketlit itself, what comes next? So we need to implement missing CRI features. For example, the attach thing that IoT-DYMux makes possible is not really working as of now. We think it was not that important, because it's usually used as a way to debug your container. But yeah, that needs to be implemented. We need to also implement this CRI container stats API. So Kubernetes uses a thing called CAdvisor to get stats from the containers. That's not a really nice code, and it doesn't work as well as we hoped. So there's a new definition in the CRI for container stats. So individual runtimes can figure out a way to get the stats and then send it back to Kubernetes. Performance is something we haven't really been looking at. But yeah, we think it's not such great performance. We see a lot of CPU usage, so we need to improve that. And as of now, Rocketlit only supports Kubernetes version 1.7, and we want to support 1.8, because that's the latest table release. We need to improve error propagation. So when a Rocket command fails, we don't give very nice error messages. Sometimes it's just an exit status. So that's something that is very high priority in our list. And we've been starting to run the conformance test for Kubernetes and CRI, and there's some tests failing, so we need to investigate those. Okay, so yeah, I want to do a demo. So at Kinfeld we developed this tool called Kupespon. And what that does is it lets you spawn multi-node clusters in your machine by using containers. So each container would be a node, basically. So instead of virtual machines, like it's usually done, like with Minikube, we use containers, and yeah, that allows us to do tests more quickly and with less resource consumption. So I have a Rocketlit-enabled Kubernetes cluster running in my machine. So a few minutes ago, I was trying to make it run, and it didn't work, but I think now it's working. So yeah, I'll share my screen. There you go. Yeah. So is this size okay? Yeah, okay. So yeah, let's see. So I have an application called MicroservicesDemo, which is something that we've worked developed. It's just a very nice microservice example application. So there's a lot of containers that work together to bring you a sock shop. So it's, yeah, a sock shop. So we can see that there's a lot of pots running for carts, the database for the carts, a catalog, payment, a lot of stuff. And yeah, I can show cluster, info, dump, grep container runtime to convince you that this is not running Docker. I can go to the machines because they're just containers. The machine is in the LCHEL. For example, this one. And yeah, there's no Docker whatsoever. There is the rocket. Yeah, this, oops, sorry, list. These are the images running. And yeah, so this is basically working. And I can get a list of the services of the namespace of the sock shop. And yeah, I can see that the front end has, no, it's an output service. So that means that if I access any IP of the cluster on this port, I will get to the front end service. So let's just try that. So yeah, I can use directly the name of the container, which is pretty cool. And yeah, this is the shop. Doesn't look so nice here in the small screen. But yeah, so you can see a catalog. So this is fully running in Rocket with Rocketlit. It's a pretty complicated application, so I think it's pretty cool. Yeah, that is basically it. So yeah, thanks. And if you have any questions, please ask. I'll try to reply. Yeah. Are there any plans to directly support the OCI format, the open container image format in Rocket? So right now we support the OCI format by being able to run Docker images, and we do a conversion there to the OCI format that's Rocket internal. And we would like to have OCI support, but I think nobody has had time yet to look at it. And people that are using Rocketlit at the moment, they use OCI images, so it's not really a priority for us now. But yeah, that would be definitely very nice to have. I think there was another question over there, but I don't know. Ah, same. So without the attach, so about the things that are not implemented that integrate with Kube, is there currently support for HPAs and things like that, and when the custom metrics come out in 1.8 for HPAs, will it support that as well? Yeah, I don't know what HPA is. Can you elaborate a bit? It's the auto scaling. The auto scaling for memory and number, not just replicas, but like the memory limits and all of that stuff, native to Kube, in 1.8 there's going to be custom metrics involved. So we see request per minute increase, and we can increase the number of processors, or the amount of the upper limits, and the initial start containers. Is Rocket going to support those as well? Yeah, I guess so. When we try to implement 1.8, we will look at those issues and we'll try to support them. It should be technically possible. Okay, thanks very much. Thank you.
 Yeah, so in this talk I'm going to talk about Rocket, what's the state of Rocket, where  we, what's being new in Rocket, and what's the future of Rocket.  And we'll also talk about Rocketlit, which is the part that integrates Rocket with Kubernetes.  So first, something's about me.  So hi, I'm Iago.  I live in Berlin, and I'm one of the Kimphol co-founders.  You probably know Kimphol by now because we are sort of organizing this conference.  So, but for those of you who don't know us, we are basically a for hire Linux engineering  team that focus on core cloud infrastructure technologies.  So yeah, one of those cloud infrastructure technologies is Rocket, and we've been working  a lot on that in the past.  And yeah, we've also retaken this work recently.  And yeah, I'm currently a maintainer of Rocket and the Rocketlit.  And yeah, there's my GitHub and my Twitter account if you want to find out more about  me.  So the plan for today is I'm going to give a brief explanation of Rocket.  So what is Rocket?  And then I will explain how it works.  So some of the Rocket internals.  I'll do the same for the Rocketlit.  What it is, how does it work?  I'll explain what's new and yeah, what are the new features we've been working on recently,  and then I will explain what's missing.  So yeah, we have a vision for the future.  So Rocket very briefly is a modern, secure, composable, container runtime.  Modern as in we try to take advantage in the latest technologies in Linux specifically  like username spaces or overlay effects or whatnot.  And come secure as in we try to be secure by default.  So all the security options are enabled by default.  If you want to disable it, you have to do it explicitly.  So you're conscious of what you're doing.  And composable because we try to play nice with the rest of the system.  We have a good integration with systemd.  It's not required, but yeah, we have a very good integration with systemd.  And yeah, composable.  And ultimately it's just an implementation of the app-c spec, specifically the parts  of image format and execution environment.  Yeah, so app-c is what we started when we started Rocket to specify the image format  and the security environment.  And now it's kind of not maintained, but Rocket still uses it heavily.  So I will give a brief look at what app-c is.  So yeah, it's a standard application container specification.  It's an open specification and it defines some associated tooling.  Yeah, there's some repos under the app-c organization like AC server or Docker2ACI.  CNI was part of the app-c spec, but now it's moved to container networking since it's  being used by a broader audience.  But for our purposes, we're going to talk a bit about the spec.  So it defines four things, image format, which is what an application consists of.  So it's basically a table with some manifest associated that says the results image you  want, executable you want to execute.  Then there's the image discovery part, which is how we translate an image name to an actual  URL where the artifact of this image is.  And for that, we take advantage of the DNS system.  So we have a translation from the name to a DNS namespace.  Then it defines spots, which is how applications are grouped and run.  And then it defines the executor, which is what the application is going to see when  it's executed.  So yeah, Rocket is a simple CLI tool.  So we don't have any demons that are mandatory.  It's written in Golang and it's basically Linux only.  And it is self-contained as in we don't require any weird dependencies, just your typical  dynamic libraries that you have installing all systems.  And it's in its system and district agnostic, although as I said, we optimize heavily for  system D.  So yeah, being a simple CLI tool, it means that the applications run directly under  the spawning process.  So there's some diagram there.  You can see that there's the spawning process, which can be bash or system D or unit, you  name it.  Then under that, there's Rocket.  And then under that, there's the pod and the two apps in this example.  So you can see these stage 0, stage 1, stage 2.  So the execution of Rocket is divided in three stages.  So basically, stage 0 is the CLI tool.  So it takes care of discovering fetching images and rendering them on the disk.  Stage 1 is basically the actual container.  So that sets up the exact environment for pods.  And it manages the processes lifecycle.  And it applies resource constraints to.  And stage 2 is the app itself.  So whatever you're running, NGINX or any app.  So one cool thing about stage 1 is that it's swappable.  So that means there's different implementations of stage 1, which is pretty cool.  So the full one uses Linux namespaces and secrets for isolation.  So what you call a Linux container.  And it's based on system D and spawn and system D.  So we basically run a system D and spawn container.  And inside, we run a system D instance that manages all the processes in the pod.  There is also a KVM implementation, which uses hardware virtualization for isolation  instead of containers.  So that's useful when you want a bit more security.  And it's based on QMU nowadays and also system D inside the pod.  So the part that manages processes is shared between those stages.  Then there's also fly, which is basically a no-isolation stage 1.  It's just a chisage route.  And its purpose is to take advantage of rockets, image handling, and signing,  and all that stuff.  There's some more, but these are the main ones.  So I'm going to stop talking about rockets.  And I'm going to introduce Kubernetes for those who don't know it.  Yeah, so basically Kubernetes is a container orchestration system.  So it gives a developer an API endpoint where it can define which containers  the developer wants to run.  And the developer doesn't have to care about where they run or how this is done.  It just says, I want to run five instances of this container.  And then Kubernetes takes care of putting those containers in actual hosts.  So you can see there that there's this funny icon on the top.  That's a CD.  And that's basically a distributed key value system  that offers the cluster like a full view of a cluster that's  consistent between among different hosts.  So that's a distributed primitive that's needed to implement Kubernetes.  And those worker kubelet boxes are basically hosts.  So on each host, a kubelet is running.  And the kubelet is the one that is responsible for creating the actual  containers inside the host.  So yeah, then the kubelet basically nowadays implements an interface called  the CRI.  Well, not implements, but defines an interface called CRI.  And CRI defines an interface where there are different methods that express  the things that you want to do with containers.  So for example, the kubelet can say Rampot Sandbox.  So a new sandbox will be started in a host.  And the kubelet can say, OK, now create this container in that sandbox  and then start it or stop it or get the status or all those stuff.  So this is implemented via gRPC, which is a remote procedure called by Google.  So kubelet will call one of those methods.  And there will be something on the other side listening and actually doing  the actual operation.  So these things are called shims.  And yeah, there are several of them.  There's CRI Continuity, which is recently been developed a lot by Google.  There's Cryo, which I think it reached version 1.0 in the last week or two weeks.  And then there's Rocketlit, which is a rocket implementation of the CRI.  And yeah, it's bold because that's what I'm going to talk about.  So how does it work?  Yeah, so you have the kubelet that connects through gRPC, as I said before,  to the Rocketlit.  And basically what the Rocketlit does when the kubelet requests a new sandbox  is called SystemD-Run, and it just starts a SystemD unit that runs Rocket,  the Rocket Sandbox.  And then the Rocketlit communicates with Rocket just running Rocket commands.  And it can ask Rocket to add containers or remove containers or whatnot.  So in order to implement this, we had to change the design of Rocket a bit  because it was basically a immutable design.  So if you start a pod, it will be this way, and that's it until you stop it  and you start a new one.  So we had to do some changes.  These changes are what we call the app experiment.  So yeah, as I said before, the CRI operates in sandbox and container terms,  not just in pods.  So Rocket had to be redesigned, and we added a new sub command for that,  which is the sub command app.  So you will have a Rocket sub command that matches each one of the methods  of this CRI interface I was talking about before.  So with this, SystemD was pretty helpful for us because this maps pretty neatly  with what SystemD does.  So for example, Rocket app add will just add a new unit file to the pod and load it.  And Rocket app start will just call SystemCTL start basically.  And yeah, this is experimental because it's a pretty big change to Rocket.  So for now, it's under an environment variable called Rocket experiment app.  So you have to specify that before calling Rocket to use these features  because if not, they will be hidden.  Another thing that we had to deal with is the logs.  So being that we run SystemD inside the pod, the natural way to handle logs  is to just use the journal.  And that's what we were doing.  Rocket was logging to JournalD.  The problem is that the CRI interface wants logs in a custom text format.  It doesn't want the journal format.  So the initial solution, kind of hackish way to solve this,  was to have basically another container on each pod that translates  the journal logs to the CRI format.  So that had some problems because it was, first of all, very resource consuming.  So you don't want to have a sidecord container.  You just want to have the containers you want to run.  And it was kind of unreliable.  So sometimes the logs were not there.  And you will get weird error messages, which nobody likes.  Moreover, we wanted to implement things like attach to the standard input  and standard output of a container.  And JournalD doesn't have anything to, so we can do that.  So the proper solution is what we call the attach experiment.  So it's another experiment.  And that uses a component called IoTTYmax.  And that allows us to write directly to the CRI file format.  So this is a simplified version of how it works.  So this is the rocket system at the rocket process tree.  So you can see systemD and spawn, and then the actual container,  which runs systemD inside.  So basically every application will connect its standard input, standard output,  and error to this IoTTYmax daemon.  And then that will pass through the standard input and output to systemD,  to an external process, or into a log file.  So in this case, only standard output and error.  So this is the simplified version of the CRI format.  It's just very simple, just to date what stream it is, and then the message.  So yeah, this allows us to attach and to write logs in the format directly  without having a sidecar like we had before.  So basically those were the two things that were the most important things  that we had to change for Rocket to work well with Kubernetes.  Yeah, I'm going to talk about the recent developments,  like in the last two, three months in Rocket.  So first we added a feature that a client requested to share the host IPC namespace,  because they wanted to do some operations on the IPC namespace.  And you need to share it with the host so other apps can actually do that.  So Michael Lee Alban implemented that.  And then we need to do a lot of fixes to make RocketLit possible.  Yeah, a lot of bug fixes, especially with the experiments.  Yeah, experimental, so they weren't really working well.  And we need to do some integration work between the attach experiment and Kubernetes,  because it needs some to put the logs in a specific path,  and we need to add some configuration options to do that.  And another thing that we did is we switched the default KVM flavor from LKVM,  which was not very reliable for us, to QM.  So yeah, now it works much better.  So yeah, what comes next in Rocket?  One thing that I think Casey started but didn't finish  is updating CNI to version 06, because we're using 03,  and there's a lot of new features in newer CNI versions that we need to update.  And we need to stabilize the Rocket experiments and remove all these environment variables.  So we need to do a lot more tests and remove those so they can be used out of a box.  Yeah, we experimented a bit with using Run C to set up the stage two runtime.  So what we were doing until now was using system.dunit files,  and those provide a lot of primitives to do a lot of resource limiting,  or mainly resource limiting.  So Run C gives a bit more flexibility though,  and we started looking at using Run C instead of,  yeah, I think Casey over there was starting to work.  Yeah, so that would be pretty cool.  And we did some experiments with CAsync,  which for those of you who don't know that,  it's a new project by Leonard Pertöding,  and I'll talk a bit more about that in the next slide,  and also general bug fixing, because there's always bugs, right?  So yeah, the CAsync tests are explained here in this graph.  So basically CAsync is a mixture between the R-Sync algorithm  and a content addressable store,  and that allows you to distribute images, for example,  in a much more efficient way,  because it will only distribute the parts that are missing.  And so we wanted to test that with container images  and see what the improvements were.  So you can see that the blue line is basically just downloading everything.  And the red line is what CAsync transfers on the network.  So you can see that there are some spikes,  and that's when a lot of things change in the image.  So in that case, you have to re-send everything basically.  But overall, you can see an improvement.  And so that's very cool.  There's the repo there, where we did the experiments,  where you can find scripts to reproduce them.  So yeah, we'll probably look a bit more into that in the future.  Yeah.  So now about Rocketlit itself, what comes next?  So we need to implement missing CRI features.  For example, the attach thing that IoT-DYMux makes possible  is not really working as of now.  We think it was not that important,  because it's usually used as a way to debug your container.  But yeah, that needs to be implemented.  We need to also implement this CRI container stats API.  So Kubernetes uses a thing called CAdvisor  to get stats from the containers.  That's not a really nice code,  and it doesn't work as well as we hoped.  So there's a new definition in the CRI for container stats.  So individual runtimes can figure out a way to get the stats  and then send it back to Kubernetes.  Performance is something we haven't really been looking at.  But yeah, we think it's not such great performance.  We see a lot of CPU usage, so we need to improve that.  And as of now, Rocketlit only supports Kubernetes version 1.7,  and we want to support 1.8, because that's the latest table release.  We need to improve error propagation.  So when a Rocket command fails, we don't give very nice error messages.  Sometimes it's just an exit status.  So that's something that is very high priority in our list.  And we've been starting to run the conformance test for Kubernetes and CRI,  and there's some tests failing, so we need to investigate those.  Okay, so yeah, I want to do a demo.  So at Kinfeld we developed this tool called Kupespon.  And what that does is it lets you spawn multi-node clusters in your machine  by using containers.  So each container would be a node, basically.  So instead of virtual machines, like it's usually done, like with Minikube,  we use containers, and yeah, that allows us to do tests more quickly  and with less resource consumption.  So I have a Rocketlit-enabled Kubernetes cluster running in my machine.  So a few minutes ago, I was trying to make it run, and it didn't work,  but I think now it's working.  So yeah, I'll share my screen.  There you go.  Yeah.  So is this size okay?  Yeah, okay.  So yeah, let's see.  So I have an application called MicroservicesDemo,  which is something that we've worked developed.  It's just a very nice microservice example application.  So there's a lot of containers that work together to bring you a sock shop.  So it's, yeah, a sock shop.  So we can see that there's a lot of pots running for carts,  the database for the carts, a catalog, payment, a lot of stuff.  And yeah, I can show cluster, info, dump, grep container runtime  to convince you that this is not running Docker.  I can go to the machines because they're just containers.  The machine is in the LCHEL.  For example, this one.  And yeah, there's no Docker whatsoever.  There is the rocket.  Yeah, this, oops, sorry, list.  These are the images running.  And yeah, so this is basically working.  And I can get a list of the services of the namespace of the sock shop.  And yeah, I can see that the front end has, no, it's an output service.  So that means that if I access any IP of the cluster on this port,  I will get to the front end service.  So let's just try that.  So yeah, I can use directly the name of the container, which is pretty cool.  And yeah, this is the shop.  Doesn't look so nice here in the small screen.  But yeah, so you can see a catalog.  So this is fully running in Rocket with Rocketlit.  It's a pretty complicated application, so I think it's pretty cool.  Yeah, that is basically it.  So yeah, thanks.  And if you have any questions, please ask.  I'll try to reply.  Yeah.  Are there any plans to directly support the OCI format,  the open container image format in Rocket?  So right now we support the OCI format by being able to run Docker images,  and we do a conversion there to the OCI format that's Rocket internal.  And we would like to have OCI support, but I think nobody has had time yet to look at it.  And people that are using Rocketlit at the moment, they use OCI images,  so it's not really a priority for us now.  But yeah, that would be definitely very nice to have.  I think there was another question over there, but I don't know.  Ah, same.  So without the attach, so about the things that are not implemented that integrate with Kube,  is there currently support for HPAs and things like that,  and when the custom metrics come out in 1.8 for HPAs, will it support that as well?  Yeah, I don't know what HPA is.  Can you elaborate a bit?  It's the auto scaling.  The auto scaling for memory and number, not just replicas, but like the memory limits and all of that stuff,  native to Kube, in 1.8 there's going to be custom metrics involved.  So we see request per minute increase, and we can increase the number of processors,  or the amount of the upper limits, and the initial start containers.  Is Rocket going to support those as well?  Yeah, I guess so.  When we try to implement 1.8, we will look at those issues and we'll try to support them.  It should be technically possible.  Okay, thanks very much.  Thank you.
2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_0_155.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_156_1751.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_1752_2447.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_2448_4871.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_4872_5939.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_5940_9875.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_9876_13547.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_13548_14759.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_14760_16439.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_16440_18563.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_18564_20447.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_20448_25787.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_25788_29495.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_29496_31463.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_31464_31691.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_31692_32339.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_32340_32387.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_32388_32543.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_32544_32627.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_32628_32759.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_32760_32879.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_32880_33143.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_33144_36947.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_36948_37007.png 2017_State_of_the_rkt_container_runtime_yTqSdn2qCIg_37008_37156.png