ML inference acceleration on K8s > with kata containers & AWS Firecracker
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Nikolos, Orestis Lagkas
The Serverless computing paradigm facilitates the use of cloud computing resources by developers without the burden of administering and maintaining infrastructure. This simplification of cloud programming appears ideal (in theory) but the catch is that when someone needs to perform a more complex task, things could get a bit more complicated. Hardware acceleration, for instance, has been a pain point, even for traditional cloud computing programming models: IaaS providers chose dedicated solutions to avoid interference and preserve tenant isolation (device passthrough), while losing one of the most important benefits of virtualization, flexibility in workload placement through live migration. Various solutions have been proposed to overcome this limitation (API remoting, hardware slicing etc.). In the Serverless world though, do we need users to interface with a hardware device directly? Most serverless deployments are backed by containers, however, the most popular (and used) one, AWS Lambda, uses a ligthweight VMM (AWS Firecracker) integrated in the container ecosystem, in order to ensure strict isolation, while maintaining scalability. To this end, enabling hardware acceleration on this kind of deployment incurs the same pain points with traditional cloud infrastructure. Kata containers evolved from clear containers and offer hypervisor support for popular orchestrators container deployments such as docker, Kubernetes etc. Through kata containers, AWS Firecracker VMs can be easily provisioned as Pods on a kubernetes system, serving workloads prepared as container images. We build on the kata container runtime and port the necessary components to support vAccel, a lightweight framework for hardware acceleration on VMs, on Firecracker. In this talk, we briefly go through vAccel, its design principles and implementation, while focusing on the integration with kata-containers and the end-to-end system applicability on ML inference workloads. We present a short patch for kata-containers to support AWS Firecracker v0.23.1, and go through the necessary patching to add the vAccel framework on k8s. Finally, we present a short demo that scales image classification purpose-built microVMs across a working K8s cluster with GPUs. Hardware acceleration for serverless deployments has never been more secure!
 Hello everyone, in this presentation we address the open problem of sharing accelerator devices on multi-terrain environments. More specifically we implement machine learning inference acceleration on Kubernetes using Cata containers running inside Firecracker VMs. Before we get there let's see the current approaches of accelerators in virtual environments. First of the first approach is called hardware partitioning and it offers the ability to partition the physical device and pass-through fragments of isolated resources to the sandbox application. Although this approach can increase resource efficiency compared to traditional full device pass-through there is still the problem of supporting different types of accelerators as it remains vendor or device specific. Another approach is to intercept API calls to vendor libraries in order to allow the use of remote hardware accelerators which means on the one hand that this is still a kind of vendor specific and on the other hand that it can lead to significant overhead due to network latency which at the end makes it a known fit for infrastructure where low latency is of importance like serverless platforms. And then there is also the split driver model where the downside is that the user needs to program the hardware directly using the vendor library and this raises concerns about the portability of an application that could be a problem especially in environments with multiple and different accelerator devices. Another concern here is that with the Paravicture Driver solution the virtual machine runs the whole vendor library in order to schedule a task in the accelerator and then the host is kind of doing the same job using the same runtime and his own scheduler. So we can say that we have something like software static application here and that could be avoided as we will show in the next slide where we introduce VXL. VXL is a universal acceleration framework that exposes generic functions to users and supports multiple acceleration backends while keeping those four design goals of simplicity, performance, portability and virtualization support. We can see here a quick overview of VXL architecture. The core component runtime system is responsible for matching the user functions with the desired acceleration frameworks as shown in the figure on the right. So the flow here is that the user application can call the user-facing API some function prototypes that at the end are intercepted by the VXL runtime and kind of float it to the desired backend. The backend talks like a hardware abstraction layer and can be either a low-level API like OpenCL or a high-level framework like TensorFlow or even a user-facing API like that's an inference. And in the more interesting case of multi-tenor environments where our workloads need to be isolated inside virtual machines, the abstraction layer can be a transport layer like VITIO. So VXL VITIO is used to afloat the function from the VM back to one of the acceleration frameworks on the host. In the current proof of concept, the VITIO acceleration mechanism can be used with QEMK VM guests and AWS Firecracker virtual machines. The core component is written in C but it also offers some bindings for us. What it is important to keep in mind here for the rest of our presentation is that at the end the VXL guest is performing IOCTL calls to a VITIO device and those calls are trapped by the kernel VITIO Excel module which leads to a trap in the virtual machine module. And then this trap is handled by the VITIO Excel backend which in the case of the QEMU is written in C and in the case of Firecracker is written in Rust. And then this VITIO backend is linked to the VXL runtime of the host which invokes one of those four acceleration frameworks. So we saw all those details about the VXL implementation and principle but now we will see how we can orchestrate our functions in an isolated virtualized environment but in the same time we can keep access to the acceleration devices on the host without having to pass through them or use the network stack or anything like that. We choose to deploy our functions on Kubernetes as Cata containers inside lightweight Firecracker virtual machines. Let's first have a look on the figure on the right so that we can have an overview of the execution flow. For simplicity we show only one application but you can think of a figure where there are multiple VMs and multiple Firecracker instances. Our application runs inside a Cata container. The Cata containers run inside the Firecracker VM and the container is started and managed by the Cata agent which also runs inside the virtual machine and is communicating with the container management system on the Kubernetes node. So essentially all the black block boxes on this figure are Cata related. The application can use the V Accelerant time API in order to ask for acceleration. This essentially means that the process from inside the container performs some IOCTL calls to the VTIO Accel device which in turn leads to the VTIO Accel kernel module. Then we have a context switch that tracks back to the Firecracker monitor which is linked to the V Accelerant time on the host and invokes the desired acceleration framework. So in order to put all these to work together we first need to patch the Cata runtime in order to support a newer version of Firecracker where the VTIO Accel back end is ported. We need it to properly configure container D in order to use the device mapper snapshotter because Firecracker uses block devices so we can currently run it only by using the device mapper. Docker was not an option here because we choose to implement these using the runtime class feature of Kubernetes which is not compatible with Docker. And as I said before the Cata agent is the one that actually creates the container from the OCI specification. So in order to expose the guest VTIO Accel device in the container we have to patch the Cata agent and append the OCI request to include the V Accel device before creating the container. Maybe you will think that maybe we could use the device or the privileged flag of the container management system but this was not working because it was probably trying to add host devices because the container this scene is running on the host while the container is invoked and created inside the guest by the Cata agent. So in any case the Cata agent needs somehow to be aware of the VTIO Accel device ID type major and minor numbers on the guest. In order to deploy these Kubernetes we install the patched version of Cata runtime in the cluster. We create the Cata Firecracker runtime class and of course configure container D and then we add some label on its Kubernetes node that supports acceleration and we also install the V Accel binaries which are essentially the Firecracker binary, the one that is linked with VTIO Accel backend, a kernel with the VTIO Accel module, a compatible RodeFS including the custom Cata agent and some configuration files. And then we make sure that the backend acceleration components are installed and working on the labeled nodes and then we can check these by using the NVDI-SMI tool as I will show in a wiring demonstration that I will make for you. Just give me a sec to open a terminal. Okay so in this demonstration I'm going to give you a quick installation overview of Cata containers with V Accel enabled Firecracker virtual machines and then we're going to deploy on our Kubernetes cluster an application that performs image classification using the V Accel framework. First we're going to make sure that there is at least one Kubernetes node labeled with V Accel. So this is the node that I'm currently logged in and we're going to use its GPU. Then we can check that there is a runtime class for Cata Firecracker runtime. The configuration is very simple. We define the name of the runtime class and the handler called CataFC which is configured in container D. Here is the name of the handler. It is configured as the runtime Cata container V2 runtime. This is the patch runtime that is compatible with the V Accel version of Firecracker. And we also have to define the configuration path for the Cata config. We can also see here that we are using device mapper as a default snapshotter for the CRI plugin. The configuration of the plugin is here. Now let's have a look at the binaries that we need to have installed on the node. The path for the Firecracker binaries is here, which is dynamically linked with V Accel library. We also have the kernel which is configured with the VTIO Accel module and the root of s with custom Cata agent that exposes the V Accel device inside the container as we show in the presentation. Now let's have a look at the actual deployment. We are going to create 36 replicas of Firecracker pods that each one will run the EMATS classification application. The application is an HTTP server that routes post requests to a handler. The handler gets an EMATS from the post body and calls the V Accel API to perform an EMATS classification operation. We also define the runtime class that will be used to run our pods. This is the Cata Firecracker runtime class that we saw before. Node selector to make sure that the pods will be deployed on V Accel compatible node. We also create a service as an endpoint to our request that will be routed to our application. So let's deploy this. So it's going to take some time. It's going to take something like a minute to create 36 Firecracker instances. They will all be created in the node that I'm currently logged in. So let's see that. You can see that here the instances have started, some of them, V20. So let's also create... They will all be created in the node that I'm currently logged in so that we can monitor the GPU processes. Let's also create an Inkers route to that service in order to send our request there. So we will send our request here and then it will be routed to the service that we created before. Okay, let's see if the deployment is ready. So we have 36 Firecracker instances running in our machine. Okay now we're going to use the NVIDIA SMI tool in order to monitor the processes that are running in our GPU. We can see that there are currently no processes running. Now let's try to send a post request containing an Inmads and see the result of the classification. You're going to get the... Let's give it a second. Okay so we're going to get an Inmads Cloud Kernels logo from Twitter and run an Inmads classification using these Inmads. And we'll send it to our application and hopefully this will be a hedgehog. Okay, let's see how we do this. We post this in the English route we created before. And normally we will see in the other terminal the GPU process that we ran for a while, then we will get the result. So we saw the process for a while in the terminal and the result was a clock. So yeah, I guess that the hedgehog kind of looks like a clock with some percentage. So now let's send a batch of requests and monitor the GPUs with a different tool, with the NV top tool. I have copied those images here from my PC. So these are the images. Let's identify the objects, the object images. Okay, we can see the GPU usage in the right terminal and also we saw the result of the classification. We'll send some more requests if let's classify the fruit images. Okay, we can see. So that's it. We can see that our application is running inside the Firecracker of Ritzel machine and offloads the computation to the GPU without having to pass through it or use their network stack. That was a very simple example of deploying machine learning inference acceleration on Kubernetes using VXL with Cata containers and Firecracker. I hope you enjoyed it. So if you find this interesting, you can give it a try. There is an installation guide in VXL website and also there is a simplified configuration to deploy these in K3S in a few minutes. You can also have a look in these two presentations, individualization and microcarnel dev rooms. The first one is focusing on VXL in Firecracker and the second one is in the microcarnel dev room and it is even more interesting as it focuses on hardware acceleration for unit kernels which I think is really, really good. So that's all. Thank you very much for watching this. Okay, so we should be live with the Q&A now. I'm not seeing any questions so far in the main dev room chat and we've had quite a bit of time actually for the Q&A on this one. So first of all, thank you very much for the talk. It was quite interesting. And yeah, we've got another 13 minutes until the next talk. So I don't know if any of you have anything that you'd like to add on top of the talk or maybe some pointers or anything else that you'd like to mention. Well, thank you, Seth. My finger, okay. Okay, I'm going to give it a little bit more time to see. There's some clapping going on, but I'm still not seeing any questions or comments at this point. Let's give it another couple of minutes. We'll wait until 25. If there's nothing to cover at that point, we can disconnect from the live chat and people can always keep asking questions in the chat. That's going to give like a 10 minutes gap until the next talk effectively. That's fine. Okay. Crickets. Seems like there's still nobody from anyone. I think we can probably wrap it up there. This room will become public at the end of the reserve Q&A period, so it's going to be taking a little while longer. It's going to go public in about 10 minutes. But until then, if anyone has any questions or any comments or anything, just ask directly in the main DevRum channel. I think one of us are in there. Actually, there's just something that came in just now. So the question is, are there special considerations for multi-node clusters with acceleration? Well, I'm not exactly sure what that means. For a multi-node cluster, you can still use the same deployment, I think. Okay, I will see if it's got any more specific questions or if that covered it. Yeah, I mean, there's a question around the recordings. So FOSDN normally has all of its videos recorded and published on fosdn.org directly accessible from the schedule. That might take a few days to show up because it needs to be merged with the Q&A and the talk itself. And then that goes through a review from the speakers to just check if there's anything that needs changing, whether we need to do some cuts before or after that kind of stuff. Once that's done, it will go live on fosdn.org. And at some point later, that gets copied over to YouTube. I know that in some years, the YouTube copy took quite a while just because of the limitations of the YouTube API in half-fashts. FOSDN can upload all the talks. There are many, many talks being recorded over the weekends that can take a little while to upload. Let's give it another minute or so to see if we get any other questions. If not, then we can wrap it up. I see someone typing, so I'll just give them a minute or a few seconds anyway. Not typing anymore. All right. Well, I think that's a wrap. Thank you very much for presenting and answering those questions. Again, if anyone has more questions, just feel free to ask in the text chat. People will be around for a while to answer those. And then in about eight minutes from now, this talk room will also open if anyone wants to join in and ask more questions. All right. Well, thanks, Jo. And maybe see you in person after another fosdn. Thank you, Stefan. Thanks. Bye-bye.
 Hello everyone, in this presentation we address the open problem of sharing accelerator devices  on multi-terrain environments.  More specifically we implement machine learning inference acceleration on Kubernetes using  Cata containers running inside Firecracker VMs.  Before we get there let's see the current approaches of accelerators in virtual environments.  First of the first approach is called hardware partitioning and it offers the ability to  partition the physical device and pass-through fragments of isolated resources to the sandbox  application.  Although this approach can increase resource efficiency compared to traditional full device  pass-through there is still the problem of supporting different types of accelerators  as it remains vendor or device specific.  Another approach is to intercept API calls to vendor libraries in order to allow the use  of remote hardware accelerators which means on the one hand that this is still a kind  of vendor specific and on the other hand that it can lead to significant overhead due to  network latency which at the end makes it a known fit for infrastructure where low latency  is of importance like serverless platforms.  And then there is also the split driver model where the downside is that the user needs to  program the hardware directly using the vendor library and this raises concerns about the  portability of an application that could be a problem especially in environments with  multiple and different accelerator devices.  Another concern here is that with the Paravicture Driver solution the virtual machine runs the  whole vendor library in order to schedule a task in the accelerator and then the host  is kind of doing the same job using the same runtime and his own scheduler.  So we can say that we have something like software static application here and that  could be avoided as we will show in the next slide where we introduce VXL.  VXL is a universal acceleration framework that exposes generic functions to users and  supports multiple acceleration backends while keeping those four design goals of simplicity,  performance, portability and virtualization support.  We can see here a quick overview of VXL architecture.  The core component runtime system is responsible for matching the user functions with the desired  acceleration frameworks as shown in the figure on the right.  So the flow here is that the user application can call the user-facing API some function  prototypes that at the end are intercepted by the VXL runtime and kind of float it to  the desired backend.  The backend talks like a hardware abstraction layer and can be either a low-level API like  OpenCL or a high-level framework like TensorFlow or even a user-facing API like that's an  inference.  And in the more interesting case of multi-tenor environments where our workloads need to be  isolated inside virtual machines, the abstraction layer can be a transport layer like VITIO.  So VXL VITIO is used to afloat the function from the VM back to one of the acceleration  frameworks on the host.  In the current proof of concept, the VITIO acceleration mechanism can be used with QEMK  VM guests and AWS Firecracker virtual machines.  The core component is written in C but it also offers some bindings for us.  What it is important to keep in mind here for the rest of our presentation is that at  the end the VXL guest is performing IOCTL calls to a VITIO device and those calls are  trapped by the kernel VITIO Excel module which leads to a trap in the virtual machine  module.  And then this trap is handled by the VITIO Excel backend which in the case of the QEMU  is written in C and in the case of Firecracker is written in Rust.  And then this VITIO backend is linked to the VXL runtime of the host which invokes one  of those four acceleration frameworks.  So we saw all those details about the VXL implementation and principle but now we will  see how we can orchestrate our functions in an isolated virtualized environment but  in the same time we can keep access to the acceleration devices on the host without having  to pass through them or use the network stack or anything like that.  We choose to deploy our functions on Kubernetes as Cata containers inside lightweight Firecracker  virtual machines.  Let's first have a look on the figure on the right so that we can have an overview of the  execution flow.  For simplicity we show only one application but you can think of a figure where there  are multiple VMs and multiple Firecracker instances.  Our application runs inside a Cata container.  The Cata containers run inside the Firecracker VM and the container is started and managed  by the Cata agent which also runs inside the virtual machine and is communicating with  the container management system on the Kubernetes node.  So essentially all the black block boxes on this figure are Cata related.  The application can use the V Accelerant time API in order to ask for acceleration.  This essentially means that the process from inside the container performs some IOCTL  calls to the VTIO Accel device which in turn leads to the VTIO Accel kernel module.  Then we have a context switch that tracks back to the Firecracker monitor which is linked  to the V Accelerant time on the host and invokes the desired acceleration framework.  So in order to put all these to work together we first need to patch the Cata runtime in  order to support a newer version of Firecracker where the VTIO Accel back end is ported.  We need it to properly configure container D in order to use the device mapper snapshotter  because Firecracker uses block devices so we can currently run it only by using the device  mapper.  Docker was not an option here because we choose to implement these using the runtime class  feature of Kubernetes which is not compatible with Docker.  And as I said before the Cata agent is the one that actually creates the container from  the OCI specification.  So in order to expose the guest VTIO Accel device in the container we have to patch the  Cata agent and append the OCI request to include the V Accel device before creating the container.  Maybe you will think that maybe we could use the device or the privileged flag of the container  management system but this was not working because it was probably trying to add host  devices because the container this scene is running on the host while the container is  invoked and created inside the guest by the Cata agent.  So in any case the Cata agent needs somehow to be aware of the VTIO Accel device ID type  major and minor numbers on the guest.  In order to deploy these Kubernetes we install the patched version of Cata runtime in the  cluster.  We create the Cata Firecracker runtime class and of course configure container D and then  we add some label on its Kubernetes node that supports acceleration and we also install  the V Accel binaries which are essentially the Firecracker binary, the one that is linked  with VTIO Accel backend, a kernel with the VTIO Accel module, a compatible RodeFS including  the custom Cata agent and some configuration files.  And then we make sure that the backend acceleration components are installed and working on the  labeled nodes and then we can check these by using the NVDI-SMI tool as I will show in  a wiring demonstration that I will make for you.  Just give me a sec to open a terminal.  Okay so in this demonstration I'm going to give you a quick installation overview of  Cata containers with V Accel enabled Firecracker virtual machines and then we're going to deploy  on our Kubernetes cluster an application that performs image classification using the V  Accel framework.  First we're going to make sure that there is at least one Kubernetes node labeled with  V Accel.  So this is the node that I'm currently logged in and we're going to use its GPU.  Then we can check that there is a runtime class for Cata Firecracker runtime.  The configuration is very simple.  We define the name of the runtime class and the handler called CataFC which is configured  in container D. Here is the name of the handler.  It is configured as the runtime Cata container V2 runtime.  This is the patch runtime that is compatible with the V Accel version of Firecracker.  And we also have to define the configuration path for the Cata config.  We can also see here that we are using device mapper as a default snapshotter for the CRI  plugin.  The configuration of the plugin is here.  Now let's have a look at the binaries that we need to have installed on the node.  The path for the Firecracker binaries is here, which is dynamically linked with V Accel library.  We also have the kernel which is configured with the VTIO Accel module and the root of  s with custom Cata agent that exposes the V Accel device inside the container as we  show in the presentation.  Now let's have a look at the actual deployment.  We are going to create 36 replicas of Firecracker pods that each one will run the EMATS classification  application.  The application is an HTTP server that routes post requests to a handler.  The handler gets an EMATS from the post body and calls the V Accel API to perform an EMATS  classification operation.  We also define the runtime class that will be used to run our pods.  This is the Cata Firecracker runtime class that we saw before.  Node selector to make sure that the pods will be deployed on V Accel compatible node.  We also create a service as an endpoint to our request that will be routed to our application.  So let's deploy this.  So it's going to take some time.  It's going to take something like a minute to create 36 Firecracker instances.  They will all be created in the node that I'm currently logged in.  So let's see that.  You can see that here the instances have started, some of them, V20.  So let's also create...  They will all be created in the node that I'm currently logged in so that we can monitor  the GPU processes.  Let's also create an Inkers route to that service in order to send our request there.  So we will send our request here and then it will be routed to the service that we created  before.  Okay, let's see if the deployment is ready.  So we have 36 Firecracker instances running in our machine.  Okay now we're going to use the NVIDIA SMI tool in order to monitor the processes that  are running in our GPU.  We can see that there are currently no processes running.  Now let's try to send a post request containing an Inmads and see the result of the classification.  You're going to get the...  Let's give it a second.  Okay so we're going to get an Inmads Cloud Kernels logo from Twitter and run an Inmads  classification using these Inmads.  And we'll send it to our application and hopefully this will be a hedgehog.  Okay, let's see how we do this.  We post this in the English route we created before.  And normally we will see in the other terminal the GPU process that we ran for a while, then  we will get the result.  So we saw the process for a while in the terminal and the result was a clock.  So yeah, I guess that the hedgehog kind of looks like a clock with some percentage.  So now let's send a batch of requests and monitor the GPUs with a different tool, with  the NV top tool.  I have copied those images here from my PC.  So these are the images.  Let's identify the objects, the object images.  Okay, we can see the GPU usage in the right terminal and also we saw the result of the  classification.  We'll send some more requests if let's classify the fruit images.  Okay, we can see.  So that's it.  We can see that our application is running inside the Firecracker of Ritzel machine and  offloads the computation to the GPU without having to pass through it or use their network  stack.  That was a very simple example of deploying machine learning inference acceleration on  Kubernetes using VXL with Cata containers and Firecracker.  I hope you enjoyed it.  So if you find this interesting, you can give it a try.  There is an installation guide in VXL website and also there is a simplified configuration  to deploy these in K3S in a few minutes.  You can also have a look in these two presentations, individualization and microcarnel dev rooms.  The first one is focusing on VXL in Firecracker and the second one is in the microcarnel dev  room and it is even more interesting as it focuses on hardware acceleration for unit  kernels which I think is really, really good.  So that's all.  Thank you very much for watching this.  Okay, so we should be live with the Q&A now.  I'm not seeing any questions so far in the main dev room chat and we've had quite a bit  of time actually for the Q&A on this one.  So first of all, thank you very much for the talk.  It was quite interesting.  And yeah, we've got another 13 minutes until the next talk.  So I don't know if any of you have anything that you'd like to add on top of the talk  or maybe some pointers or anything else that you'd like to mention.  Well, thank you, Seth.  My finger, okay.  Okay, I'm going to give it a little bit more time to see.  There's some clapping going on, but I'm still not seeing any questions or comments at this  point.  Let's give it another couple of minutes.  We'll wait until 25.  If there's nothing to cover at that point, we can disconnect from the live chat and people  can always keep asking questions in the chat.  That's going to give like a 10 minutes gap until the next talk effectively.  That's fine.  Okay.  Crickets.  Seems like there's still nobody from anyone.  I think we can probably wrap it up there.  This room will become public at the end of the reserve Q&A period, so it's going to  be taking a little while longer.  It's going to go public in about 10 minutes.  But until then, if anyone has any questions or any comments or anything, just ask directly  in the main DevRum channel.  I think one of us are in there.  Actually, there's just something that came in just now.  So the question is, are there special considerations for multi-node clusters with acceleration?  Well, I'm not exactly sure what that means.  For a multi-node cluster, you can still use the same deployment, I think.  Okay, I will see if it's got any more specific questions or if that covered it.  Yeah, I mean, there's a question around the recordings.  So FOSDN normally has all of its videos recorded and published on fosdn.org directly accessible  from the schedule.  That might take a few days to show up because it needs to be merged with the Q&A and the  talk itself.  And then that goes through a review from the speakers to just check if there's anything  that needs changing, whether we need to do some cuts before or after that kind of stuff.  Once that's done, it will go live on fosdn.org.  And at some point later, that gets copied over to YouTube.  I know that in some years, the YouTube copy took quite a while just because of the limitations  of the YouTube API in half-fashts.  FOSDN can upload all the talks.  There are many, many talks being recorded over the weekends that can take a little while  to upload.  Let's give it another minute or so to see if we get any other questions.  If not, then we can wrap it up.  I see someone typing, so I'll just give them a minute or a few seconds anyway.  Not typing anymore.  All right.  Well, I think that's a wrap.  Thank you very much for presenting and answering those questions.  Again, if anyone has more questions, just feel free to ask in the text chat.  People will be around for a while to answer those.  And then in about eight minutes from now, this talk room will also open if anyone wants  to join in and ask more questions.  All right.  Well, thanks, Jo.  And maybe see you in person after another fosdn.  Thank you, Stefan.  Thanks.  Bye-bye.
containers_ml_kata_0_179.png containers_ml_kata_180_539.png containers_ml_kata_540_671.png containers_ml_kata_672_3695.png containers_ml_kata_3696_6875.png containers_ml_kata_6876_12275.png containers_ml_kata_12276_13763.png containers_ml_kata_13764_16127.png containers_ml_kata_16128_17255.png containers_ml_kata_17256_17507.png containers_ml_kata_17508_17615.png containers_ml_kata_17616_18335.png containers_ml_kata_18336_18755.png containers_ml_kata_18756_21095.png containers_ml_kata_21096_23363.png containers_ml_kata_23364_24647.png containers_ml_kata_24648_24947.png containers_ml_kata_24948_27407.png containers_ml_kata_27408_27455.png containers_ml_kata_27456_27599.png containers_ml_kata_27600_27659.png containers_ml_kata_27660_27707.png containers_ml_kata_27708_27827.png containers_ml_kata_27828_28787.png containers_ml_kata_28788_29915.png containers_ml_kata_29916_31739.png containers_ml_kata_31740_31787.png containers_ml_kata_31788_31871.png containers_ml_kata_31872_33407.png containers_ml_kata_33408_37379.png containers_ml_kata_37380_38411.png containers_ml_kata_38412_38927.png containers_ml_kata_38928_39083.png containers_ml_kata_39084_39299.png containers_ml_kata_39300_41471.png containers_ml_kata_41472_41585.png