HPC Container Engines: State-of-the-Art
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kniep, Christian
The Container ecosystem spans from spawning a process into an isolated and constrained region of the kernel at bottom layer, building and distributing images just above to discussions on how to schedule a fleet of containers around the world at the very top. While the top layers get all the attention and buzz, this session will base-line the audiences' understanding of how to execute containers. The talk will briefly recap the history of containers and containers in HPC and describe the challenges we faced and how the community overcame them; eventually converging towards a sustainable model to run HPC application at scale using containers. By attending this sessions viewers will come to understand that HPC runtime engines are not far apart anymore and what to look out for when adopting HPC containers in 2021. We'll also poke at the next layer up - Image build and distribution - to describe the challenge after we picked the runtime engine.
 Hey there, my name is Christian Knieb. I'm a developer advocate for HPC at AWS. A while back I gave a talk about containers at FOSSTEM 2016. Today I would like to baseline everyone on HPC runtimes and engines so that we are all on the same page and have a common understanding about how to dissect the container ecosystem. This talk will firstly recap why containers are a good thing in general while explaining why the standard tech was not flying in HPC. Second, I will discuss why that is in particular. With that we are going to dissect the container ecosystem into palatable slices. Fourth, from change route all the way to modern HPC runtimes. In conclusion, I'll recap and touch on what is the next layer to be built on top of runtimes. When DOCAF was announced and released in 2013, it started the widespread adoption of containerization on developer laptops. Open research sites like NERSC or CSCS started to get requests to run containers on their HPC systems soon after. But with the DOCAF architecture and stack at the time, it was impossible to be applied to the shared environment HPC sites are running in. In the early days, running containers was used to run a service in isolation. Let us have a look at the monolithic stack from back then. We had a client running in the user context, communicating with the same binary running as a service in root context. The process running within the container is a child of the actual DOCA demon. Even worth, you were able to trick the kernel's POSIX access barrier. The container API allows to define the user within the container and bind mount directories. Without SELinux or App-a-More watching over file system access, it was a scary thing and it seems like Pandora's box for system admins. Also for traditional list, who stands was, why do we need this in the first place? We are fine with what we have. But let's be clear. This is the intended use of the API until today. In non-HPC land, that is fine. By keeping the container independent of the host and data access managed in a more cloudy fashion, not relying on the POSIX barrier, rather whether a volume is available in the first place, that's also fine. All of that did not hold back students and researchers knocking on doors to ask how to run containers in open research facilities. Thus, forks at NERSC, Barclay and LANL started experimenting. Hey, Charlotte Gleut, shifter and singularity. Before we dive deeper into the runtimes, let us first discuss the challenges we face in HPC or rather in shared systems through the lens of iterations of machine learning. As they were the first to pick up container tech. Okay, first we start with the simplest version. We use TensorFlow on a node in total isolation. The data is located in a volume only accessible by the container. We download it within the container from S3 even. We only access CPU, memory and network through the container. That is how containers were designed to work. Please keep this in mind. Total portability as we do not make assumptions about the underlying host. Build, ship and run everywhere is the slogan and the abstraction of the underlying host made sure it worked. But containers were just too attractive to not bundle up all the weird stacks used in more progressive and hard to control stacks like AI ML. Thus, they were quickly adopted to access GPUs. Access to the driver and some devices and off you went. NVIDIA was quick to jump all in and help by forking Docker as NVIDIA Docker. Which was hacky at first but over the years became a good citizen utilizing standards. Data was still isolated though. As we saw in my scary example earlier, the screaming began when you needed access to a shared POSIX file system without control over the UID of the process. By then you also encountered the first cracks in your understanding of the clear cut between user land, kernel space drivers. But you blew the lid off completely when you were dealing with distributed apps using kernel bypassing NICs, MPI and traditional HPC schedulers. MPI run needs to reach out to remote host, typically using SSH and we don't want SSH to run in all containers, right? PMI relaxes this a tad but you still need to initialize the MPI domain across all instances. To wire up all the processes, MPI communication does not like nutted networks typically used in container land and needs to communicate with other processes. All of that with the added complexity of MPI with its different variants, ABI compatibility and PMI. Today we as a community are pretty much able to address all those challenges. But back then skeptics wanted all or nothing. I'd say the scariest one was the POSIX issue because it was the easiest to understand. The rest was WD40 and duct tape. It was easy to find edge cases to poke holes and break the current setup. Some just threw in a security expert that stalled the process for sure. With the challenges in mind, let us dissect the ecosystem from the bottom up. The lowest layer is the runtime. Its sole purpose is to create an actual interactive process with some isolation, at least an isolated fire system view for the process. The easiest and earliest example of a runtime might be change root. What is needed? The root file system we can change into, and I will fetch an Alpine table here. To change root, we need to have privileges, which makes sense if you think about what you can do. You can create your own ETC pass VD, change the pseudo file. Change root is pretty much the keys to the kingdom. But still, it created a process within a new root file system. Fast forward to today. We can use the same root file system to spawn a container using RunC. We already got the root file system from the change root example. Only thing missing is the runtime spec. We will create one using RunC spec, and we are even creating a rootless container. No privileges needed anymore. In running the container, we have a similar look and feel as with change root. But the configuration created applies some sensible defaults. The user within the container is mapped to an unprivileged user outside of the container. Access to devices is filtered, syscalls are limited, and so forth. RunC is the first implementation of an OCI compliant runtime. Back when monolithic Docker stack was broken up into models for reuse. But RunC for sure is the plumbing. The user needs to come up with the configuration file. It won't deal with fetching images and extracting root file system for later use. That is why a layer above is needed. After exploring the runtime, let us look at the engine. It's taking care of all the setup. Assembling the config and dealing with downloading an image, optimized for distribution. After the image is downloaded, it will extract the root FS, what is called a snapshot, which a runtime will use. In the RunC example, we used Wget and tar to download the image and extract the snapshot and RunC spec to generate a configuration file. We could also use Docker to download an image, extract the snapshot, and copy the snapshot over to a local directory. In the traditional Docker stack, container D is used as an engine. And don't worry, we will have a look at other engines like Seros and Potman as well. Now let us have a look how this engine and runtime plays out in the Docker stack first. This architecture is pretty busy with all the interaction options and plugins. Let me reduce the noise so that we can focus on the important parts. We are using the Docker CLI to interact with Docker D. And I'm a bit over simplifying here. I'm ignoring the difference between content and image, ignoring the namespace service and the task service. But for this talk, we will go with only three pieces. Images, snapshots, and Run. Okay. In Dockerland, we use Docker pull to pull an OCI image from a registry. For pull, we reach out to the registry for the latest image available and check the local store if it is already present. If not, we download the image and keep it locally. Docker create does what we saw me doing in the example. Snapshot the image into a root file system and eventually create a configuration. Docker start takes the final step to launch the container using a runtime. And all of the three steps are done when you use Docker run, even though it won't reach out to the registry if the image name is already present. It just trusts the image. Now we follow the flow tracing the process of a Docker run command. The Docker client runs in the user context and reaches out to the Docker service, a system service. Docker D context the container D to supervise the actual container creation on the host. To allow for better life cycle management like logging and easier error handling, the shim is forked for each container. The shim uses run C to create a container, but only for exactly that. After the container process runs, run C will die and the container process is reparented to the shim. If we get back to our challenges overview, this is fine as long as we don't do not break too much of our abstraction. Next the GPUs is done via runtime called OCI hook. I'm going to dive deeper into OCI hooks later in the talk when we discuss Seros. Due to the use of the root context, we are still in scary territory with regards to shared POSIX file systems. Even though in cloudy environments, the issue is partly solved by running ephemeral clusters per user, getting rid of the shared in shared file systems. The distributed applications still have problems as the actual container process is not really a direct child of what the client fired up, along with the problems of network and inter-process communication I talked about earlier. MPI run, Docker run, just is not going to happen easily. Okay, I assume we all agree that container services like Docker do not cut it. We need to go rootless and keep everything in user land. Let's first have a look at Podman. Red Hat advertises it as a replacement for the Docker stack, as it implements the same CLI interface without a system service in the middle. Calling Podman run will directly fork a shim, same goal as a container dshim, better lifecycle and error and lock handling. This shim called container monitor uses RunC to create a container, which again dies afterwards to hand over the responsibility to Conman. The accelerator challenge is handled with OCI hooks. Since we are running in the user context, we cannot trick the kernel by posing as a different UID. The team around Podman also makes sure to play nice with MPI run, including logic within Podman. Let's look at the early HPC run times, which are alive and kicking to this day. Charlie Cloud, shifter and singularity. They were created back in the early days, and their focus was to leverage the packaging of containerized software in HPC environments. Shifters focus was to provide a working environment for their HPC site. They created a lifecycle, which allowed a user to request a Docker image to be pulled into the HPC site via an image gateway. And snapshotted this onto a shared file system for a user to use. The runtime was handcrafted and is going to be similar to what we discussed in this section. Let's look at Charlie Cloud. Charlie Cloud uses exec to start a process and reparent the process to the bash of the user. Since it is a user-land container, we do not have the issue of POSIX breakout, and it uses external scripts on the host to deal with HPC customization of the container. Using the mechanism of but not quite being OCI hooks. For singularity, familiar approaches are taken. The starter SUID will be the parent of the process. It is able to hook in GPUs and deal with MBI. Again, not quite using OCI hooks, but using similar approaches baked into singularity. Singularity also provides options to use namespaces like the pit namespace. I had a little fight over this with Greg from Singularity when I invited him to speak at Docker HQ back in the days. Are those early HPC runtimes real containers? I say when we got started, it was a stretch as most were just using the mount namespace to provide a dedicated file system. But gradually, more namespaces were added. Even if they are not true OCI runtimes, as they does not implement the full runtimes spec, they for sure have their place in the HPC community. Personally, I prefer the integrated approach of using standard container components and specifications as it leads to better adoption and interoperability. That brings us to my favorite runtime, as it is using standards and components everywhere. Serous. Serous directly uses RunC to start a container. Et voilà. No shim. Since Serous has the HPC focus, it does not need to compromise and worry about long-running services like Docker and potentially Podman, as it aims to appeal to non-HPC use cases. Serous uses the TrottenPars for GPU acceleration and OCI hook. Since it is a user-land container, no worries in terms of POSIX file system. Like the rest of the pack, pure OCI hookjoy when we get to the real HPC nitty-gritty. So let's dive into that a little bit more. Let's get started with a slide from the Serous team at CSCS. They gave this in 2020's high-performance container workshop. The actual lifecycle event got an update which deprecated pre-starts in favor of create runtime. If we follow the lifecycle from create to destroy container, we can see the different stages are used to manage the lifecycle of a different aspect. For example, a resource might need to check which libraries are present within the container file system. This can be checked at create runtime before the actual container process is started. Post start might be used to signal that the container process has just started and post stop can be used to free up resources for others to use. For reference, the different events defined in the runtime specification. Create runtime to customize the container after its namespaces are created. Create container to customize the container after the file system is set up. Start container is used just before the container is being started. For instance, to inspect the libraries used by the binary. Post start to send signals just after the process was forked and post stop to clean up or debug after the container is deleted. This is how the actual hooks are configured within the runtime specification. Pretty straightforward, I think. I'll pause for a bit to let you check the code. Looking at the hooks that CEROS already provides. The standard NVIDIA GPU hook to allow for GPU use. An API hook that ensures the correct replacement of MPI libraries from the host. G-Lip C to replace the container for the host one as the mounted MPI dependency might rely on the newer ones. An SSH hook to inject a key and an SSHD into the container. The slurm hook that waits for all processes to be started before executing the containerized application. And finally, the timestamp hook to assist debugging the container lifecycle. To conclude this breakout, some examples of hook configuration. The GPU hook might be configured like this on a given instance. I will pause a little bit so that you can grasp it. The MPI hook for Mvpage 2 with the condition via annotation. Again pause a little bit. Now some final thoughts on that. Hooks externalize runtime decisions. Configured on the actual host, the container is starting on. This means that the configuration is left to the system administrator and thus is transparent to the user. Otherwise we would need to deal with some of this within the container. We might wait for a condition, even a job-wide one, to be met before starting the container on each host. The beauty of this for the user is that he can trust runtime decisions are made on a host itself. And the system administrator is in charge of making sure certain assumptions are met. And there is certainly more things we can do with this hook. Fix an infrastructure need without a hack script. Fetch data before the container starts. Fix file permissions along the container lifecycle. I think there are much more things that we can think of. This concludes the exploration of the lower layers. Once we are all on the same page, even if that means we agree to disagree, we can look beyond the execution. Scheduling and workflow orchestration is going to be a fun one. Who is in charge of execution? Slurm cluster on top of Kubernetes. Kubernetes within Slurm. Airflow driving Slurm supervised by a Jupyter notebook. And as another threat, how do we build and distribute images? My humble opinion, spec becomes pretty appealing to create container images from the receipts we use to install natively. But we need to get smarter when it comes to distributing optimized images, taking the context of the node and job into account. Let's free ourselves from runtime decisions. Okay, that's all we have time for today. Let's recap this session. Firstly, I would like to encourage everyone to split the concerns in terms of runtime and engine. I had so many discussions over the past almost seven years in which those two concepts were mixed and the discussion went nowhere because everyone was talking about different aspects and prerequisites. That is the reason I'm happy with where we are at as a community. We are explicitly reusing common runtimes like RunC and not reinventing the wheel as we as a community were guilty of so many times. Secondly, please do not fall into the trap of confusing how we distribute images and how we snapshot and store them for immediate use. If you are a user at an HPC site, you might have to file a ticket to get an image from the interweb to become a snapshot for you to use. Similar to what shifter pioneered with the image gateway. But that does not mean that you should not care about how to share the OCI images across multiple sites. That would need to adhere to standards, I think. Lastly, please get out of your echo chamber and explore different options. It seems like we are all human and the runtime we are socialized with is the one we keep using no matter what. That is fine, but it does not hurt to explore and understand the foreign cons of each runtime in different contexts. By the time this goes live, hopefully I have the time to create this workshop you see on the slides. The goal is to provide an environment to explore all the runtimes out there using a common example code. And if you have questions, please reach out. A good place is the HPC container Slack community. But feel free to ping me an email to have a chat about the options and how you and your R can explore what is best for them. Thank you for taking the time to join me on this session.
 Hey there, my name is Christian Knieb.  I'm a developer advocate for HPC at AWS.  A while back I gave a talk about containers at FOSSTEM 2016.  Today I would like to baseline everyone on HPC runtimes and engines so that we are all  on the same page and have a common understanding about how to dissect the container ecosystem.  This talk will firstly recap why containers are a good thing in general while explaining  why the standard tech was not flying in HPC.  Second, I will discuss why that is in particular.  With that we are going to dissect the container ecosystem into palatable slices.  Fourth, from change route all the way to modern HPC runtimes.  In conclusion, I'll recap and touch on what is the next layer to be built on top of runtimes.  When DOCAF was announced and released in 2013, it started the widespread adoption of containerization  on developer laptops.  Open research sites like NERSC or CSCS started to get requests to run containers on their  HPC systems soon after.  But with the DOCAF architecture and stack at the time, it was impossible to be applied  to the shared environment HPC sites are running in.  In the early days, running containers was used to run a service in isolation.  Let us have a look at the monolithic stack from back then.  We had a client running in the user context, communicating with the same binary running  as a service in root context.  The process running within the container is a child of the actual DOCA demon.  Even worth, you were able to trick the kernel's POSIX access barrier.  The container API allows to define the user within the container and bind mount directories.  Without SELinux or App-a-More watching over file system access, it was a scary thing and  it seems like Pandora's box for system admins.  Also for traditional list, who stands was, why do we need this in the first place?  We are fine with what we have.  But let's be clear.  This is the intended use of the API until today.  In non-HPC land, that is fine.  By keeping the container independent of the host and data access managed in a more cloudy  fashion, not relying on the POSIX barrier, rather whether a volume is available in the  first place, that's also fine.  All of that did not hold back students and researchers knocking on doors to ask how to  run containers in open research facilities.  Thus, forks at NERSC, Barclay and LANL started experimenting.  Hey, Charlotte Gleut, shifter and singularity.  Before we dive deeper into the runtimes, let us first discuss the challenges we face in  HPC or rather in shared systems through the lens of iterations of machine learning.  As they were the first to pick up container tech.  Okay, first we start with the simplest version.  We use TensorFlow on a node in total isolation.  The data is located in a volume only accessible by the container.  We download it within the container from S3 even.  We only access CPU, memory and network through the container.  That is how containers were designed to work.  Please keep this in mind.  Total portability as we do not make assumptions about the underlying host.  Build, ship and run everywhere is the slogan and the abstraction of the underlying host  made sure it worked.  But containers were just too attractive to not bundle up all the weird stacks used in  more progressive and hard to control stacks like AI ML.  Thus, they were quickly adopted to access GPUs.  Access to the driver and some devices and off you went.  NVIDIA was quick to jump all in and help by forking Docker as NVIDIA Docker.  Which was hacky at first but over the years became a good citizen utilizing standards.  Data was still isolated though.  As we saw in my scary example earlier, the screaming began when you needed access to  a shared POSIX file system without control over the UID of the process.  By then you also encountered the first cracks in your understanding of the clear cut between  user land, kernel space drivers.  But you blew the lid off completely when you were dealing with distributed apps using kernel  bypassing NICs, MPI and traditional HPC schedulers.  MPI run needs to reach out to remote host, typically using SSH and we don't want SSH  to run in all containers, right?  PMI relaxes this a tad but you still need to initialize the MPI domain across all instances.  To wire up all the processes, MPI communication does not like nutted networks typically used  in container land and needs to communicate with other processes.  All of that with the added complexity of MPI with its different variants, ABI compatibility  and PMI.  Today we as a community are pretty much able to address all those challenges.  But back then skeptics wanted all or nothing.  I'd say the scariest one was the POSIX issue because it was the easiest to understand.  The rest was WD40 and duct tape.  It was easy to find edge cases to poke holes and break the current setup.  Some just threw in a security expert that stalled the process for sure.  With the challenges in mind, let us dissect the ecosystem from the bottom up.  The lowest layer is the runtime.  Its sole purpose is to create an actual interactive process with some isolation, at least an isolated  fire system view for the process.  The easiest and earliest example of a runtime might be change root.  What is needed?  The root file system we can change into, and I will fetch an Alpine table here.  To change root, we need to have privileges, which makes sense if you think about what  you can do.  You can create your own ETC pass VD, change the pseudo file.  Change root is pretty much the keys to the kingdom.  But still, it created a process within a new root file system.  Fast forward to today.  We can use the same root file system to spawn a container using RunC.  We already got the root file system from the change root example.  Only thing missing is the runtime spec.  We will create one using RunC spec, and we are even creating a rootless container.  No privileges needed anymore.  In running the container, we have a similar look and feel as with change root.  But the configuration created applies some sensible defaults.  The user within the container is mapped to an unprivileged user outside of the container.  Access to devices is filtered, syscalls are limited, and so forth.  RunC is the first implementation of an OCI compliant runtime.  Back when monolithic Docker stack was broken up into models for reuse.  But RunC for sure is the plumbing.  The user needs to come up with the configuration file.  It won't deal with fetching images and extracting root file system for later use.  That is why a layer above is needed.  After exploring the runtime, let us look at the engine.  It's taking care of all the setup.  Assembling the config and dealing with downloading an image, optimized for distribution.  After the image is downloaded, it will extract the root FS, what is called a snapshot, which  a runtime will use.  In the RunC example, we used Wget and tar to download the image and extract the snapshot  and RunC spec to generate a configuration file.  We could also use Docker to download an image, extract the snapshot, and copy the snapshot  over to a local directory.  In the traditional Docker stack, container D is used as an engine.  And don't worry, we will have a look at other engines like Seros and Potman as well.  Now let us have a look how this engine and runtime plays out in the Docker stack first.  This architecture is pretty busy with all the interaction options and plugins.  Let me reduce the noise so that we can focus on the important parts.  We are using the Docker CLI to interact with Docker D. And I'm a bit over simplifying here.  I'm ignoring the difference between content and image, ignoring the namespace service  and the task service.  But for this talk, we will go with only three pieces.  Images, snapshots, and Run.  Okay.  In Dockerland, we use Docker pull to pull an OCI image from a registry.  For pull, we reach out to the registry for the latest image available and check the local  store if it is already present.  If not, we download the image and keep it locally.  Docker create does what we saw me doing in the example.  Snapshot the image into a root file system and eventually create a configuration.  Docker start takes the final step to launch the container using a runtime.  And all of the three steps are done when you use Docker run, even though it won't reach  out to the registry if the image name is already present.  It just trusts the image.  Now we follow the flow tracing the process of a Docker run command.  The Docker client runs in the user context and reaches out to the Docker service, a system  service.  Docker D context the container D to supervise the actual container creation on the host.  To allow for better life cycle management like logging and easier error handling, the  shim is forked for each container.  The shim uses run C to create a container, but only for exactly that.  After the container process runs, run C will die and the container process is reparented  to the shim.  If we get back to our challenges overview, this is fine as long as we don't do not break  too much of our abstraction.  Next the GPUs is done via runtime called OCI hook.  I'm going to dive deeper into OCI hooks later in the talk when we discuss Seros.  Due to the use of the root context, we are still in scary territory with regards to shared  POSIX file systems.  Even though in cloudy environments, the issue is partly solved by running ephemeral clusters  per user, getting rid of the shared in shared file systems.  The distributed applications still have problems as the actual container process is not really  a direct child of what the client fired up, along with the problems of network and inter-process  communication I talked about earlier.  MPI run, Docker run, just is not going to happen easily.  Okay, I assume we all agree that container services like Docker do not cut it.  We need to go rootless and keep everything in user land.  Let's first have a look at Podman.  Red Hat advertises it as a replacement for the Docker stack, as it implements the same  CLI interface without a system service in the middle.  Calling Podman run will directly fork a shim, same goal as a container dshim, better lifecycle  and error and lock handling.  This shim called container monitor uses RunC to create a container, which again dies afterwards  to hand over the responsibility to Conman.  The accelerator challenge is handled with OCI hooks.  Since we are running in the user context, we cannot trick the kernel by posing as a  different UID.  The team around Podman also makes sure to play nice with MPI run, including logic within  Podman.  Let's look at the early HPC run times, which are alive and kicking to this day.  Charlie Cloud, shifter and singularity.  They were created back in the early days, and their focus was to leverage the packaging  of containerized software in HPC environments.  Shifters focus was to provide a working environment for their HPC site.  They created a lifecycle, which allowed a user to request a Docker image to be pulled  into the HPC site via an image gateway.  And snapshotted this onto a shared file system for a user to use.  The runtime was handcrafted and is going to be similar to what we discussed in this section.  Let's look at Charlie Cloud.  Charlie Cloud uses exec to start a process and reparent the process to the bash of the  user.  Since it is a user-land container, we do not have the issue of POSIX breakout, and it uses  external scripts on the host to deal with HPC customization of the container.  Using the mechanism of but not quite being OCI hooks.  For singularity, familiar approaches are taken.  The starter SUID will be the parent of the process.  It is able to hook in GPUs and deal with MBI.  Again, not quite using OCI hooks, but using similar approaches baked into singularity.  Singularity also provides options to use namespaces like the pit namespace.  I had a little fight over this with Greg from Singularity when I invited him to speak at  Docker HQ back in the days.  Are those early HPC runtimes real containers?  I say when we got started, it was a stretch as most were just using the mount namespace  to provide a dedicated file system.  But gradually, more namespaces were added.  Even if they are not true OCI runtimes, as they does not implement the full runtimes  spec, they for sure have their place in the HPC community.  Personally, I prefer the integrated approach of using standard container components and  specifications as it leads to better adoption and interoperability.  That brings us to my favorite runtime, as it is using standards and components everywhere.  Serous.  Serous directly uses RunC to start a container.  Et voilà.  No shim.  Since Serous has the HPC focus, it does not need to compromise and worry about long-running  services like Docker and potentially Podman, as it aims to appeal to non-HPC use cases.  Serous uses the TrottenPars for GPU acceleration and OCI hook.  Since it is a user-land container, no worries in terms of POSIX file system.  Like the rest of the pack, pure OCI hookjoy when we get to the real HPC nitty-gritty.  So let's dive into that a little bit more.  Let's get started with a slide from the Serous team at CSCS.  They gave this in 2020's high-performance container workshop.  The actual lifecycle event got an update which deprecated pre-starts in favor of create runtime.  If we follow the lifecycle from create to destroy container, we can see the different  stages are used to manage the lifecycle of a different aspect.  For example, a resource might need to check which libraries are present within the container  file system.  This can be checked at create runtime before the actual container process is started.  Post start might be used to signal that the container process has just started and post  stop can be used to free up resources for others to use.  For reference, the different events defined in the runtime specification.  Create runtime to customize the container after its namespaces are created.  Create container to customize the container after the file system is set up.  Start container is used just before the container is being started.  For instance, to inspect the libraries used by the binary.  Post start to send signals just after the process was forked and post stop to clean up or debug  after the container is deleted.  This is how the actual hooks are configured within the runtime specification.  Pretty straightforward, I think.  I'll pause for a bit to let you check the code.  Looking at the hooks that CEROS already provides.  The standard NVIDIA GPU hook to allow for GPU use.  An API hook that ensures the correct replacement of MPI libraries from the host.  G-Lip C to replace the container for the host one as the mounted MPI dependency might rely  on the newer ones.  An SSH hook to inject a key and an SSHD into the container.  The slurm hook that waits for all processes to be started before executing the containerized  application.  And finally, the timestamp hook to assist debugging the container lifecycle.  To conclude this breakout, some examples of hook configuration.  The GPU hook might be configured like this on a given instance.  I will pause a little bit so that you can grasp it.  The MPI hook for Mvpage 2 with the condition via annotation.  Again pause a little bit.  Now some final thoughts on that.  Hooks externalize runtime decisions.  Configured on the actual host, the container is starting on.  This means that the configuration is left to the system administrator and thus is transparent  to the user.  Otherwise we would need to deal with some of this within the container.  We might wait for a condition, even a job-wide one, to be met before starting the container  on each host.  The beauty of this for the user is that he can trust runtime decisions are made on a  host itself.  And the system administrator is in charge of making sure certain assumptions are met.  And there is certainly more things we can do with this hook.  Fix an infrastructure need without a hack script.  Fetch data before the container starts.  Fix file permissions along the container lifecycle.  I think there are much more things that we can think of.  This concludes the exploration of the lower layers.  Once we are all on the same page, even if that means we agree to disagree, we can look  beyond the execution.  Scheduling and workflow orchestration is going to be a fun one.  Who is in charge of execution?  Slurm cluster on top of Kubernetes.  Kubernetes within Slurm.  Airflow driving Slurm supervised by a Jupyter notebook.  And as another threat, how do we build and distribute images?  My humble opinion, spec becomes pretty appealing to create container images from the receipts  we use to install natively.  But we need to get smarter when it comes to distributing optimized images, taking the  context of the node and job into account.  Let's free ourselves from runtime decisions.  Okay, that's all we have time for today.  Let's recap this session.  Firstly, I would like to encourage everyone to split the concerns in terms of runtime  and engine.  I had so many discussions over the past almost seven years in which those two concepts were  mixed and the discussion went nowhere because everyone was talking about different aspects  and prerequisites.  That is the reason I'm happy with where we are at as a community.  We are explicitly reusing common runtimes like RunC and not reinventing the wheel as  we as a community were guilty of so many times.  Secondly, please do not fall into the trap of confusing how we distribute images and  how we snapshot and store them for immediate use.  If you are a user at an HPC site, you might have to file a ticket to get an image from  the interweb to become a snapshot for you to use.  Similar to what shifter pioneered with the image gateway.  But that does not mean that you should not care about how to share the OCI images across  multiple sites.  That would need to adhere to standards, I think.  Lastly, please get out of your echo chamber and explore different options.  It seems like we are all human and the runtime we are socialized with is the one we keep  using no matter what.  That is fine, but it does not hurt to explore and understand the foreign cons of each runtime  in different contexts.  By the time this goes live, hopefully I have the time to create this workshop you see on  the slides.  The goal is to provide an environment to explore all the runtimes out there using a common  example code.  And if you have questions, please reach out.  A good place is the HPC container Slack community.  But feel free to ping me an email to have a chat about the options and how you and your  R can explore what is best for them.  Thank you for taking the time to join me on this session.
hpc_container_engine_state_of_art_0_179.png hpc_container_engine_state_of_art_180_1391.png hpc_container_engine_state_of_art_1392_3851.png hpc_container_engine_state_of_art_3852_4271.png hpc_container_engine_state_of_art_4272_5663.png hpc_container_engine_state_of_art_5664_8603.png hpc_container_engine_state_of_art_8604_8891.png hpc_container_engine_state_of_art_8892_8999.png hpc_container_engine_state_of_art_9000_9047.png hpc_container_engine_state_of_art_9048_9179.png hpc_container_engine_state_of_art_9180_9623.png hpc_container_engine_state_of_art_9624_11195.png hpc_container_engine_state_of_art_11196_11747.png hpc_container_engine_state_of_art_11748_12047.png hpc_container_engine_state_of_art_12048_12251.png hpc_container_engine_state_of_art_12252_12599.png hpc_container_engine_state_of_art_12600_14411.png hpc_container_engine_state_of_art_14412_14531.png hpc_container_engine_state_of_art_14532_14615.png hpc_container_engine_state_of_art_14616_18203.png hpc_container_engine_state_of_art_18204_20543.png hpc_container_engine_state_of_art_20544_22679.png hpc_container_engine_state_of_art_22680_23915.png hpc_container_engine_state_of_art_23916_24779.png hpc_container_engine_state_of_art_24780_25235.png hpc_container_engine_state_of_art_25236_26339.png hpc_container_engine_state_of_art_26340_26387.png hpc_container_engine_state_of_art_26388_26687.png hpc_container_engine_state_of_art_26688_26735.png hpc_container_engine_state_of_art_26736_27011.png hpc_container_engine_state_of_art_27012_28403.png hpc_container_engine_state_of_art_28404_29627.png hpc_container_engine_state_of_art_29628_32639.png hpc_container_engine_state_of_art_32640_33043.png