Replacing Docker with Podman
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Walsh, Dan
This talk will describe all of the reasons for podman, all of its features demonstrate its functionality, I will cover the background of podman, how we built it, why we built it, I will demonstrate using it in multiple different ways, Running containers building container images Communicating with it via var link, cockpit integration. Communicating with it from a remote machine.
 So my name is Dan Walsh. I work for Red Hat. I actually, my official title is lead, well, my job role is a consulting engineer at Red Hat. My title is lead architect of the run times team at Red Hat. So we handle everything underneath Kubernetes and underneath OpenShift needed to run containers. So we have a whole bunch of people working on our teams and we've been developing a whole bunch of, really we should call them container engines. The run time term is overused. Really what these are is container engines and I see the low things like run C and Cata containers is really being the run times. So really these things are container engines. So this talk is called replacing Docker with Podman. Really what we, our view of the world is actually replacing tearing apart what Docker did into a series of sub components. So Podman really is replacing the Docker CLI. So it's sort of the traditional way you run Docker commands is what we're looking for for, with the Podman command. So first you, the DNF install, I could have put AppGet up here, your DNF install Podman. Then you do this. Any questions? All right. And to show you that's true, this guy, Alan Moran, who I don't know, a couple months ago said, I completely forgot that two months ago I set up an alias of Docker equals Podman and it has been a dream. No big fat demons, project atomic. Down below, one of the comments to him was, how did you figure out you were using Podman instead of Docker? And he said, I did, I did Podman help, I mean Docker help and it came up and gave him Podman's help message. So that's how we figured it out. So obviously I can't stop at that. So at this point, everybody has to stand up. If you've seen me do talks before, I make you guys participate. Okay. So please read out loud anything that is in red. Excellent. Thanks work. All right. So containers is a Linux thing or it's basically a concept. And do you guys go make copiers and say I'm going to make a Xerox? Or do you take a tissue paper out and say it's a Kleenex? Or do you take an aspirin out and say it's an aspirin? Oh, that's a bad point. But basically, at this conference, I cringe in the back of the room every time I hear someone basically use the D word. And I actually have another talk where I have a swear job that I put up and every time I say the D word, I have to put money into the swear job. But I'm not going to do that today because I don't have much money. But anyways, what do you need to run a container? Okay, what does it mean when I run a container? And the first thing you need when you want to run a container, you need to identify what the hell is a container? And a container, in this case, has been standardized or at least the image format or what people mean when they say I'm going to run a container. And they're talking about something that sits at a container registry, like Docker I.O. or Quay.io, or there's probably 100 different people out of factory who are all doing container registries. And there's these images or the tab balls that sit up there, container images. And a couple of years ago, thanks to CoreOS, if you saw the talk earlier on Vincent's talk, talked about the history of container run times, CoreOS introduced the app C spec, caused the fracture, all of a sudden there were going to be two different types of container images. I mean, it actually forced all of the companies, the big companies that are involved in containers like, big companies and startups, to get together and say, we're going to standardize on what this, what it means to be a container image. And that's where OCI came out. And the big companies, I'm talking about Red Hat, IBM, Google, Microsoft, Docker, CoreOS at the time, who we've now acquired, as you most of you know, got together and they standardized on what it means to be a container image last year, last December actually, they came out with the OCI image bundle specification. And now we sort of have a good idea of what it means to be a container. So I say, I want to run the Fedora container, I know what I'm going to get when I pull it down, or have an idea of what I'm going to get. So the next thing I need to do is get a mechanism for pulling images, pulling the images off of container registry to the host. And again, some of this was covered earlier this morning, but basically we built a tool several years ago, actually Antonio in the back of the room did, to, we actually did a pull request upstream because we found is that people were pulling images off of container registries and they were huge. I mean, we're talking some images of 1.5 gigabytes, two gigabytes, right? These are huge images. And it's a JSON file that basically describes what's in the image. So what we said is why don't we build a command like darker inspect dash dash remote to pull down the JSON file associated with the image that I could then look at and then figure out if I actually want to pull down the image, right? The only way to get an image and look at what's inside of it right now is to pull it to the host. You have to pull that two gigabytes to your machine before you say, oh, that's not really what I wanted. Let me get rid of it. So we went upstream with that pull request and they said, sorry, we're not interested. It confuses the API or the CLI too much. But they said, but it's just a web interface, right? These are just container registries. Container registries are nothing but web services with tie balls on it. So it's all web protocol. So they said, go off and build your own tool to go out and pull down the JSON. Look at it with your own tool. So we built Scopio. Scopio means remote viewing in Greek. And that's why we have a Greek hat and a telescope. And so after we built Scopio, Antonio actually went off and he started implementing more of the protocols for pulling registries. So instead of just pulling down the JSON, he also pulled down the image. And he also figured out he could use Scopio to push images. And Scopio basically slowly evolved into this really cool tool that you can start to move images around the environment. And you don't have to be rude to do this. So you can actually copy off of one registry, copy to another registry without ever having pulled the image bundle to your host. So it really evolved into a cool tool. And we were actually working with CoreOS before we acquired them. And we were trying to convince them to use Scopio to move images in and out of rocket. And they said, well, we don't really want to be exerting a tool. Why don't you make it into a library? So we created a containers image. So containers image is a library that now is working independently. Now people are contributing lots and lots of pull requests into to be able to move images around. So it's really a base level image. The number one contributor outside of Red Hat is actually Pivotal, who's one of Red Hat's biggest competitors in the open ship space. But Pivotal is using basically containers image for moving images and out of, I think they call it garden, is their source for that. So the next thing after you pull the image to the host, you need to be able to explode the image onto disk. And usually in a Linux world or in a container world, we call it, we use copy and write file systems. Because basically an image tends to be a layered thing. So you basically want to install a layer of an image. Then you're going to create another type of file system or another mount point on top of it. You put the second layer and you put another layer on top. And to do that, you use copy and write file systems. So way back when we were first starting to work with Docker, Red Hat, actually Alex, if he's here somewhere, did most of the work, the guy that's in charge of Flatpak now, basically introduced a whole bunch of different types of copy and write file systems into what was Docker at the time. That was overlayFS, butterFS, device mapper. And so what we did is we took all that code and we moved it into an independent library. So it was independent from basically the upstream Docker project. And then we went and started to evolve that library. So all these things are independent. Next thing you need to do, so you could basically define an image. We pull the image to the host. We store it on top of some kind of storage. And then we need a standard mechanism means what does it mean to run a container? Well, luckily, OCI standardized on that. So the OCI specification was a runtime, the second specification was called a runtime specification, which basically says, I'm going to write a JSON file. Everybody has to understand what that JSON file looks like. And then I need to basically launch a program that reads that JSON file and creates the container on the system, all the container processes, sets up the C groups, security settings, namespaces. So that's the last part of running a container on the system. One other thing we needed is we needed sort of a monitor. When I'm running a container, a container can just exit. So when a container just exits, it doesn't know it's running in a container. It's just a process on the system. So you need something to actually watch that process. If the process exits, you want to grab its exit code and store it somewhere. OK, put it on the system. You actually want to keep open the TTYs that are connected to it because people are going to come to you and say, hey, what's going on inside of that container? So you need a process that sits out there and sort of monitors it. And that's called conmon. So if you went to the earlier talk by Antonio, he talked a little bit about conmon and how it's used inside a cryo. So conmon is a simple C program that basically is the parent of the container. So it can parent to the PID1 inside the container and just sits out there running till the container exits, catches a sig child, and then it basically stores away some last minute stuff and exits. That means that any one of these container engines that we're going to be talking about, like cryo and like podman, can go away. They don't have to stay running on top of the container to watch what's going on. So there's a little conmon process out there running. Did I skip ahead? Oh, yeah. I just explained conmon and the CNI was up there. Sorry about that. Someone should have said that. So CNI is basically a way for, it's an interface, again, introduced by CoROS, that basically defines a network protocol for the container engines to set up networking. So other people can come in and basically get a plug-in interface to plug in different kinds. We're going to be using it in the cloud, we're going to be using it inside the Kubernetes, we're using it inside a cryo, we're going to be using it inside a podman. So basically all different types of people, tooling can basically build a CNI plug-in and then we can use it with these tools. So basically we have the four building, five, six building blocks here that allow us to experiment with different types of container engines. And we're not talking about a container engine. That is something that you talk to and say, pull me an image. So I know what an image is, pulls it down to the disk, puts it on to storage, configures the run-c specification or the OCI specification and then launches the runtime. Saves data like what happens when the container exits and reports it back to the human. So it's basically a human interface or the tooling interface for running containers. So one of my problems with Docker is it has become a big fat container demon. So it's become basically a roadblock for innovation. Having to have a demon to basically launch process on a Linux system just seems to be wrong. Everybody that runs the Docker CLI thinks that the container is a child process of the client. What's actually happening is the Docker program client is talking out to the server, talking out to a demon, and actually the process that gets launched as PID 1 ends up being a child or a grandchild of the Docker demon, not of your process that launches. I'm going to show you some interesting things about that. But what's also happened is if you have only one way of doing containers, it stops all innovation. If I want to do some special things, if I want to move those container images around, I have to go to the one entity and say, may I please do this? And they say, that's not really of interest in this upstream project. So you get denied. So by breaking it apart, we basically get the best of both worlds. All different tools can use, basically can contribute to these different components, and all of a sudden you can start to build some interesting tools on top of it. So this talk is talking about Podman. So Podman means does everybody here know what a pod is? In Kubernetes world, Kubernetes launches pods. They don't launch containers. What a pod is, is a process on the system that has one or more containers inside of it. So what Kubernetes wanted is they want to launch these multiple, one or more processes all locked together in the same namespaces and then be able to move them around the system. People came the other night, there's been some talks that talked about these sidecar containers. So you might have your primary application and you might lock into it another container and that second container is basically monitoring the first container, basically figuring out whether or not that container is or if it's doing something on behalf of that container. I think someone was talking the other night about basically it does all the authorization for it, so the application, the primary application can do stuff and not have to worry about authorization. It's just a sidecar container that does it. So Kubernetes wanted this concept where I could run more than one container at the same time around the environment and it just manages pods. So when we built Podman, Podman is part of our libpod effort, which is basically a library to build pods and so we wanted to build Podman as a tool for managing pods and containers in the environment. But what we didn't want to do when we built Podman was basically give you a brand new UI or CLI. So we wanted to basically, we started out by copying the Docker CLI. So to run commands inside of Podman, it basically uses pretty much the exact same CLI that you use when you run Docker. So if you find any Docker command in the world, theoretically you should be able to just substitute Podman for it. So lastly, before I get to the demo, we show this is the architecture. Again, this is the same architecture that we showed earlier for the same picture that was shown earlier for a cryo. So we have, when I'm running a pod, I have a series of commands. So when I'm running Podman, it's going to go out and create this environment. So I have a command that runs, so if I'm running a pod directly, I will have an infracontainer that basically just holds open all of these namespaces and C groups and then I have one or more containers and running inside of it. So Podman can run pods, but it also can run regular containers. So the traditional way you run containers. So at this point, we're going to demo it. By the way, the icon here, a group of CLs is called a pod. So that's where we got the... So first, we're just going to do a Podman version. We're going to be running sudo to run it as root and of course, like any good security engineer, I don't have sudo without password. So here we go. We just launched this as version 0.9 of Podman. It's all written in go because that's what the cool kids want to write in. And what the reason it's 090 is this was... Well, we've been releasing Podman on a weekly basis. So it's not 1.0 yet and the 9 stands for the month and the 1 stands for the week. So this is the first week of September version of Podman. And so I'm going to show you info to give you... So it's using container storage. Containers are stored under ViLive containers. We basically have some additional features up above. You probably... I scrolled off the screen, but I'm not going to go up because I'll probably screw it up. But it's running on top of the overlay file system. And some neat things that we've added is actually... This is something different than Docker. From a security point of view, we actually mount all the devices with no dev by default. So even the images and that's showing you that you can pass in special overlays. And this is all stuff that's built into container storage has given us all sorts of new features that we can take advantage of. So I'm going to cat out a Docker file. So we're about to... This is a Docker file that I used when I'm about to demonstrate right now is actually Podman running a container that has Builder, which is... Builder is another one of our projects, which is a container. It's going to run a Builder and build a Docker file, build based on the top of a Docker file inside of a container without giving any privileges to it. So here we go with the demo gods, hopefully this... So it's going to pull down Alpine image because that's the smallest image available. If you look at it, we're actually volume mounting in here. I have a MyVol directory and volume mounting it in. I'm actually using SC Linux, so we're changing the label on it. And then I have my containers, I'm bounding that into my live containers. So Builder inside of the container in this case is going to be writing to a host directory on the system. And then I'm using a VFS, so I've changed the type of the storage and there it is. So it's now finished. It actually built a container image with no big fat demons. So there's no demons running on the system. There's nothing up my sleeves or whatever. So there we're in inside of a container, another container, inside of a lockdown container that basically built an image. Now that image could actually be pushed to a container registry all without any special privileges. So you can imagine when we talked about Cryo earlier, you can actually use tools like Podman and Builder inside of containers that are locked down. So you can actually run really interesting workloads inside of a Kubernetes distribution. This kind of shows it. So here I'm going to actually show the image that was just built. So that's interesting. It doesn't enjoy that. But here we have, I pulled down an Alpine image. It actually had a couple of layers that got installed and then basically it ended up creating my image inside of it. So clear that screen. So interesting thing here is one of the things I get asked often when I'm supposedly a security guy, when I go to customers, customers are all worried about their engineers that are coming to them all the time and saying, I got to build this Docker thing. I got to build this Docker thing. And I have to get access to the Docker socket. So the Docker socket is running its root. And you can actually do things like Docker run dash, dash privilege, dash v slash colon slash host, Fedora, root slash host, and boom, I have full root on the system. It's actually worse. Giving someone access to the Docker socket is worse than giving them sudo without a password because there's no logging. As soon as I screw around on your machine, I can go and blow away that container. And as soon as I blow away that container, all the logging gets eliminated from the system. So what they want to do is they want to be able to use the Docker CLI on the system without requiring root. So here we are. I'm about to show you running podman without root. So I'm going to pull an image. And again, hopefully the network stays up. Everybody that's doing a YAM update on the system, please stop right now. So it's going to pull down an image to the system. Do this very fast, but this network is. Talk amongst yourselves. Right. So this is podman running without root. There is no daemons out there. So what happens when I run podman as a non-root user, it actually creates the storage. So instead of the storage being out on ViLive containers, it actually ends up being in home directory.local slash share slash containers slash storage, I believe. So actually I'm going to show you that I just pulled down an image to my system. And just for that shows you the images on the host. So there's a lot more images on the host. There's only one image in my home directory. If I want to run a container on top of, so there I just ran alpine container, ran the ls command inside of a container in my home directory. No root needed. So what are we doing to cause this? So how are we actually doing this? We're taking advantage of the user namespace, which most of you have never seen before or at least had very little exposure to. So we actually have a tool on the builder called builder on share. I just want to show you, it basically puts you into a user namespace without being inside of a container. So I am, right now I'm logged in as de-wash on the system. That process right now that's showing you the root on the home directory, that's de-wash. Okay, so it basically did a swap. So what happens in the system is this is actually a program going on here. So if I did ID right now, you will see as far as it's concerned, I am running as root on the system. If I did a cat of proc UID map, you would see the mapping that's going on inside of the user namespace. So if you're logged on to Fedora system and probably Ubuntu has the same thing, shadow utils now puts every user logs into the system. It puts an entry into Etsy sub UID and Etsy sub GID that defines the mapping that is available to this user on the system. So my system, you see as I said my UID is 3267, it maps UID 0 to that. And it said there was one mapping in the range. Then it says I am going to map UID 1 to 100,000 and I am going to do it to 500,000 UIDs. So that means in my home directory now, I can map 501,000 UIDs. I can create UIDs from my own UID as well as 100,000, 100,000, 100,000, 100,000, 100,000, 200,000 to all the way up. And those would be mapped to one to 500,000 inside of the container. Pretty cool, huh? It leads to some interesting problems, though. I can create content in my home directory that I can't delete after I can do it unless I'm in the username space. If I exit the username space back to my regular UID, I have files in my home directory that I can no longer delete. So I actually just wrote a blog on this past week. Okay, so Podman has some interesting username space. So username space has always been this nirvana for container isolation. I just showed you how you can use it in a home directory, but what would be really nice is if I could use it on the system for separating containers. So if I run a, you know, right now, when I run Docker or Podman, I'm using different SE Linux labels for each one of them. That gives me a lot of time to do that. So I'm going to run this in my home directory. I'm going to run different SE Linux labels for each one of them. That gives me isolation. With username space, I want to be able to isolate, basically, have this container, have this range of UIDs, have this container, have this range of UIDs, and therefore, if I broke out of the container, UID 5000 would not be able to interact with UID 6000 on the system, right, just like normal UID separation. The problem is the reason we've never used this, using the name space predateStalker. But up to this point, there's never been a file system support for this. So username space has been really, really cool, but no one's ever used it. So what we did with Podman is actually we built, since the file system doesn't support it, what we've done is we're using, we're actually choning files underneath the covers to be able to make username space work. So here I'm going to create a username space for UID 0 to, well, starting at 100,000 to 500,000. That just created one container. Now I'm going to actually look at it. So one of the other things we've done with, actually, guys from Suzy did this, this work, when I run a container, the Docker had this command called DockerTop to show you the process is running inside the container. So we've actually used this new library called PSGo that actually can do something really cool. We can actually show you the UID inside of the container, as well as the UID outside of the container. And this is a brand new tool, a brand new enhancement to Podman that basically allows you to see that inside the container it says I'm user root, but outside of the container, if I actually looked at that process, the pit on that process is running as 100,000. Matter of fact, I'll show you that right now. So here we see the sleep program that I launched inside of the container is actually running as 100,000. So now I'm going to run another container, and this time, instead of using 100,000 for the first process, I'm going to use 200,000 for the first process. So now I have the container, and if I look at that container, I'd see that I'm running as root inside of the container, I'm running as root outside of the container, and if I look in the system, I will see that one sleep is running as 100,000, and the other one is running as 200,000. But from inside of the container, they both think they're root. What's happening on the file system underneath, we've taken the Alpine file system, we're actually churning it on the fly, and we have some really interesting tools that we're adding to make that churning faster and better, all different. So I talked earlier about Docker being this client-server model where you exec a program on your file system, and it talks to a server, and that causes lots of issues. It causes things that we can't do like things like SD Notify. Everybody knows what SystemD does with SD Notify. Basically, you're going to run a process inside of a container, and it's going to call back to SystemD and say, I'm ready to receive requests. Well, that never worked inside of Docker because you can't get the SD Notify. You put the Docker command inside of a SystemD Unifile, it's talking to the Docker daemon away on a different socket, and the process is saying, I'm ready to Docker. It's not saying it back to the Docker client, it's saying back to the Docker server. So what I'm going to show you is Podman actually does exactly what you think it does. So the way I do that is, does anybody know what this login UID is? When you log onto a Linux system, what happens is there's a UID, part of your process system, records that you were Dan Walsh, right, that you were logged into the 3267. So this will show this on the system. It basically says, I logged into 3267. I can't change that. I can become root. I can do anything on the system, sudo anything else. This login UID tracks me. There's no way for me to get, change that login UID once it's set. That means that the auditing subsystem can record the fact that I did something, whether I was root, a different user, or anybody else, but it was Dan Walsh that did it. So here we're going to run a container. So we're running a container, and the container is basically showing you, inside of the container process right now, it's running as login UID Dan Walsh, right? It's running as 3267. If I run Docker, it shows that it's running as UID, that happens to be UID minus one. Okay? Because that's an init system, Docker, daemon never logged onto the system. So if I do something evil on the system, it comes back, via Docker, and it comes back and says, Docker did something evil. If I do it through Podman, it's going to come back and say, Dan Walsh did something evil. So what I'm going to do here is I'm putting a watch on Etsy Shadow, which hopefully worked. You didn't even know it showed that? Oh, it says the rule already existed, which is good, because I ran this earlier. And now inside of Podman, I am touching, I am trying to rewrite Etsy Shadow, right? And if I do this down here, I got trapped. So it shows that AUID, D Walsh, did something to Etsy Shadow. Now if I do it through Docker, and it shows AUID on set did something to Etsy Shadow. So what that's demonstrating from a security point of view, Podman executing a command basically tracks what the user did on the system, as opposed to Docker. And this is why I say when you give people access to the Docker socket, it's more powerful than sudo, because if I go through sudo and I do something in your system, the audit system knows that Dan Walsh did it. So it showed a little bit of Podman top features. I'm going to run a container. I can see the SC Linux label via Podman top. I can see this is something that no one sees when they run Docker. No one has any idea. These are the Linux capabilities that are on by default when I run a Podman. So I've shown people are always asking, what capability? Everybody talks about capabilities. What should I drop? What should I add? Well, these are the default lists that you get in almost every container you run in the environment. If you run TRIO, we run with a lot less. A lot of these capabilities are there just to be able to build containers. Okay, make node. That's so I can create device nodes. If I'm running containers underneath TRIO in production, I'm not expecting people to build device nodes. So we take it away by default. You don't get make node. We don't get rid of it when we're running it. But basically, there's different... If you look at running applications, there's different ways of running applications. In this case, we need more privileges because we don't know what you're going to do with Podman. So we call a Podman for a reason. One of the things we want to do is be able to manage pods. What I've shown you so far is basically all the CLI that matches up with Docker. But here we have pods. So if I want to create a pod, I'm going to use Podman pod. I'm going to name the pod. And it's going to create a pod in the system. That actually... The echo line there is wrong. I didn't fix that, but don't worry about it. I'll make this all available and hopefully clean it up. So Podman, now I'm going to actually create another pod. I'm going to create a sleep pod. But this time, I'm going to tell it... I'm going to create a container, but I'm going to put the container inside of the pod. Okay, so this is basically creating a pod. So I created a pod. Now I'm assigning a container to the pod. I'm going to create another container and assign it to the pod. And guess what? I'm going to show on the system. Okay, these are all the containers I ran earlier. But basically, right now, it would have worked a lot better. I should have killed all the containers. But basically, I am about to start the pod. The two containers I just created did not create containers. So at that point, now you should see two more containers. So you've seen containers that just got second, got created. They were created on the system. But basically, they just status, they got up one second ago. So you can see, when I started the pod now, I started both of those containers. If I want to stop the pod... Now this is actually a bug. When I'm stopping a container, it waits 10 seconds for the container to actually exit. If the tool catches that and sleeps actually catching it, there's a bug in Podman right now that it actually waits on the first one before it sends the signals to the second one. So we actually have to fix that so it sends the signals to all of them. So it's going to wait about 20 seconds. So there, we did it, and we should be back to three containers running on the system. And again, it would be better if I killed all the containers before I ran it. But basically, it's running. So that shows you pods running on the system. If I want to remove all the pods, I can actually force it to remove all the containers that are created for the pod. And now if I list out the pods, I'm back to zero pods running on the system. So that is a real quick demonstration of some of the features that you can do with Podman. So let's talk about other things. I mentioned some of this stuff earlier, since a lot of people here are interested in SystemD. Over the years, I came to SystemD a couple of years ago, and I gave a talk about the CTO of Docker versus Leonard, and both of them were being very hard on working together. So one of the things I did when we built Podman is we wanted to have proper SystemD integration into it. So proper, you can just run a container that has SystemD, and it will just work under Podman without any modifications. We support, as I talked about with login UID and demonstrated, we actually can do proper SD notify. So we run a container, the container does SD notify, that triggers all the way up into Podman. Podman reports a SystemD that the container is up and running and ready to receive actions. We also do socket activation, so you can actually put Podman into the system and have SystemD automatically fire it up, and it will pass down the socket down to the process that's running on the system. So all the things we wanted to do with Podman is we wanted to have some kind of, right now it's written in Go, we wanted to have an interface to it that people could use other than Go, so we added a remote API for it. So we wanted to basically say, well, I can run Podman, I can set up a SystemD socket-activated Podman instance, and then have an API that talks to it. So we actually decided to use Violink. So Violink is an API communication tool, a library that we can use to actually do socket activation and communicate to a Podman running on the system, and it allows you to socket activation of Podman. And here's a unit file that we provide, so if you install Podman, you will get a Podman, it's actually IO Podman service file, create a socket file and a service file. If you enable those, then you can start to run remote commands with Podman. So here we have a Python, we also provided a Python library that you can build Python programs to communicate with containers. So you can do an input, this is showing importing a Podman and it's listing, I think it's dumping information. So this is dumping the information about Podman on the host, but basically it's a full API that you can communicate with Python, via Python to it. We also have, so we built a program called PyPodman that does all of the Podman commands, but it's already in a Python, and the Python command talks to the Violink, through Violink to the server, and so you can do all of the Python. Why do we write Python command to do this? Because we wanted to do Python on different operating systems. So here is a demonstration of using PyPodman. Hopefully this, you're not going to be able to see it, but... Basically it's going to run all the commands on the system, so it's basically doing PyPodman there to list the containers. It's showing you some of the config information, showing info, that's the Podman info command that I wrote. So this is doing it all, and rather than, since most of you people can't see it, the last step of this is it actually shows this is all running on top of a Mac. So this is running a Python script on top of a Mac that's talking to a virtual machine that's running Violink into a Podman client sitting in a system D unit file. So we built this protocol to be able to talk from the Mac to the server we called it SSH. So we're taking advantage of SSH on a Mac, or on a Windows box, or a remote Linux box, to actually talk to Violink over the system. So all you have to do to make this work is actually use SSH, set up SSH, to communicate between the two boxes, and it'll work. We're adding cockpit support since we have clients, so this is for stuff. So we're adding Podman support into cockpit. Sadly, I have it running on here, and it didn't show any images, so we're not going to show you that. But basically, I have someone, Martin Pitt's been working to help us get this working. But basically, we're getting full integration between cockpit and Podman, and in this case, we're using Node.js to actually talk to the Violink protocol to Podman running in a system D unit file. Now, I say no big fat demons, I'm kind of cheating a little bit. But the thing here is that Podman is just running for the connection that happens. So we're firing up multiple Podmans for every single connection, so we're not doing any kind of monitoring of multiple containers or anything like that. So what don't we do? What we don't do? Right now, so I've talked about us replacing Docker, but the stuff that the Docker demon did that we're not doing inside of Podman. So one of the things we don't do is we don't do autorestart, because there's this thing called system D that does a real good job of autorestart. So if you want a container that's going to be restarted if it fails, it blows up, you put Podman inside of your unit file, and it just works. We don't do Docker swarm. We work on Kubernetes. If you want to orchestrate lots and lots of containers, use cryo on top of Kubernetes. We don't do notary right now, but we'd be very willing, if someone wants to open up patches to make Podman work with notary, we'd be very willing to do it. Red Hat's probably not going to put the engineering into doing that. We don't do health checks yet. Health checks could probably be done as a sidecar container, or they could probably be done as a system D unit file, or like a unit file we create. Basically, a health check is supposed to come up periodically and make sure that the container is running properly on the system, but we haven't quite figured out how we're going to do about that. We don't do the Docker API. So if you have tools that talk the Docker API to the Docker demon, we don't have a tool for that. Lastly, we don't do Docker volumes yet. So Docker volumes, we do most of what you think of regular volumes, file system volumes, but there is people that have built Docker, demons for the Docker demon that do volumes. We're looking at doing that very soon. So that's planned on the roadmap to be able to support that. This point, this is the real point where we ask questions. Anybody have questions? Yes. A lot of people have questions. We're just going to hang around. So first thing in terms of the implementation of Docker API, there is one reason for doing that. Maybe not in Podman, but we can just provide some wrapper on top of the var link API. Because without that, I don't have integration in all of the IDEs, IntelliJ, whatever. And without that, I am unable to get rid of the Docker from the developers' workstation. So you're talking about twist lock and stuff like that? No, no, no. I mean the idea from the IntelliJ, or PyCharm, or whatever, or Visual Studio Code, every IDE for the developer has integration. Is all talking from a root process, launching wrapper root process. We'd be willing to accept, if people wanted to build something like that on it, it probably would not be hugely complicated. I understand. It's not the scope of the Podman, but it's just an idea of wrapper. That's one thing. We get rid of the Docker on almost all places, except one Docker file. So Docker file is, we support the Docker file format. I'll sign up for a lightning talk later, and I'll talk about Builder. So I can do a 5-minute on Builder. Right now, Podman only supports Docker file format. Builder supports anything you want. For the Builder, I perfectly know that I can just write simple shell script and create whatever I want. But not everyone should write shell scripts. The goal of Builder is to basically allow you to build another tool, like Ansible Container, or Source to Image, and other people to build tools that don't have to generate a Docker file in order to build an image. But it would be nice to have one standard. Let's let other people question that. I'll talk to you forever outside. Okay. Just yell. Does Podman support a union of file systems like OverlayFS? Yes, it runs until right now. It supports OverlayFS. We have a bug on DeviceMapper. So it supports OverlayFS by default. We have a bug in DeviceMapper right now. DeviceMapper was built under the concept that only one process would do it. But we have ideas of how we're going to fix it. It runs on top of ButterFS, and I don't know if anybody's tested it. We also have VFS, so VFS layer. Someone's built an Overlay file system for... So when you're running in a non-root environment, it's actually using VFS, because you can't use OverlayFS on that. But one of my engineers wrote a thing called FuseOverlayFS that actually works real well for non-root. So we're actually introducing that, too. Anybody else? Yeah. You got to wait. Oh, sorry. One, two? Okay. One of the reasons I've still been using D mostly for decompose? Yeah, so Docker compose. I didn't put that one up. Seems like you could... We've been asked about that, and again, I'd be willing to accept it. We're thinking of... That's on the roadmap to do some kind of compose, whether or not we support the Docker compose language, or we support a Kubernetes compose language, or we support an OpenShift compose language. Again, Partman's a fully open-source project, so anybody that wants to contribute is welcome to contribute, and we will take patches for that, but right now we don't have compose yet. Anybody else? Oh, up here. I mean, it's actually plus one for the API, because Ansible runs via API. Then the word service API. So without that... So Ansible? Yeah. Well, I mean, you mean Ansible container or something? Ansible container. Yeah, Ansible container, they're looking into moving directly to Builda for support for that. So to basically get out of the Docker team. Right now, in my opinion, everybody's basically put the Docker socket all over the place. The Docker socket is one of the most dangerous things you can do. As soon as you give the Docker socket, you give full root to your system without having any tracking, and most of the time you're doing that to do Docker builds. Right? So usually, you're basically giving your people the ability to create a Docker file and then build it into an image. Well, I just showed you that you can run... In fact, you can run Partman non-root with Builda inside of a container and actually build the Docker image and push it to Docker I.O. I just said Docker image, so I owe money. Okay? That container image. I blew it. I think that's the first time I really screwed up. Okay? Anybody else? All right. I'm out of time anyways. I've been... Thank you.
 So my name is Dan Walsh.  I work for Red Hat.  I actually, my official title is lead, well, my job role is a consulting engineer at Red  Hat.  My title is lead architect of the run times team at Red Hat.  So we handle everything underneath Kubernetes and underneath OpenShift needed to run containers.  So we have a whole bunch of people working on our teams and we've been developing a whole  bunch of, really we should call them container engines.  The run time term is overused.  Really what these are is container engines and I see the low things like run C and Cata  containers is really being the run times.  So really these things are container engines.  So this talk is called replacing Docker with Podman.  Really what we, our view of the world is actually replacing tearing apart what Docker  did into a series of sub components.  So Podman really is replacing the Docker CLI.  So it's sort of the traditional way you run Docker commands is what we're looking for  for, with the Podman command.  So first you, the DNF install, I could have put AppGet up here, your DNF install Podman.  Then you do this.  Any questions?  All right.  And to show you that's true, this guy, Alan Moran, who I don't know, a couple months ago  said, I completely forgot that two months ago I set up an alias of Docker equals Podman  and it has been a dream.  No big fat demons, project atomic.  Down below, one of the comments to him was, how did you figure out you were using Podman  instead of Docker?  And he said, I did, I did Podman help, I mean Docker help and it came up and gave him Podman's  help message.  So that's how we figured it out.  So obviously I can't stop at that.  So at this point, everybody has to stand up.  If you've seen me do talks before, I make you guys participate.  Okay.  So please read out loud anything that is in red.  Excellent.  Thanks work.  All right.  So containers is a Linux thing or it's basically a concept.  And do you guys go make copiers and say I'm going to make a Xerox?  Or do you take a tissue paper out and say it's a Kleenex?  Or do you take an aspirin out and say it's an aspirin?  Oh, that's a bad point.  But basically, at this conference, I cringe in the back of the room every time I hear  someone basically use the D word.  And I actually have another talk where I have a swear job that I put up and every time I  say the D word, I have to put money into the swear job.  But I'm not going to do that today because I don't have much money.  But anyways, what do you need to run a container?  Okay, what does it mean when I run a container?  And the first thing you need when you want to run a container, you need to identify what  the hell is a container?  And a container, in this case, has been standardized or at least the image format or what people  mean when they say I'm going to run a container.  And they're talking about something that sits at a container registry, like Docker I.O.  or Quay.io, or there's probably 100 different people out of factory who are all doing container  registries.  And there's these images or the tab balls that sit up there, container images.  And a couple of years ago, thanks to CoreOS, if you saw the talk earlier on Vincent's talk,  talked about the history of container run times, CoreOS introduced the app C spec, caused  the fracture, all of a sudden there were going to be two different types of container  images.  I mean, it actually forced all of the companies, the big companies that are involved in containers  like, big companies and startups, to get together and say, we're going to standardize  on what this, what it means to be a container image.  And that's where OCI came out.  And the big companies, I'm talking about Red Hat, IBM, Google, Microsoft, Docker, CoreOS  at the time, who we've now acquired, as you most of you know, got together and they standardized  on what it means to be a container image last year, last December actually, they came out  with the OCI image bundle specification.  And now we sort of have a good idea of what it means to be a container.  So I say, I want to run the Fedora container, I know what I'm going to get when I pull it  down, or have an idea of what I'm going to get.  So the next thing I need to do is get a mechanism for pulling images, pulling the images off  of container registry to the host.  And again, some of this was covered earlier this morning, but basically we built a tool  several years ago, actually Antonio in the back of the room did, to, we actually did  a pull request upstream because we found is that people were pulling images off of container  registries and they were huge.  I mean, we're talking some images of 1.5 gigabytes, two gigabytes, right?  These are huge images.  And it's a JSON file that basically describes what's in the image.  So what we said is why don't we build a command like darker inspect dash dash remote to pull  down the JSON file associated with the image that I could then look at and then figure  out if I actually want to pull down the image, right?  The only way to get an image and look at what's inside of it right now is to pull it to the  host.  You have to pull that two gigabytes to your machine before you say, oh, that's not really  what I wanted.  Let me get rid of it.  So we went upstream with that pull request and they said, sorry, we're not interested.  It confuses the API or the CLI too much.  But they said, but it's just a web interface, right?  These are just container registries.  Container registries are nothing but web services with tie balls on it.  So it's all web protocol.  So they said, go off and build your own tool to go out and pull down the JSON.  Look at it with your own tool.  So we built Scopio.  Scopio means remote viewing in Greek.  And that's why we have a Greek hat and a telescope.  And so after we built Scopio, Antonio actually went off and he started implementing more  of the protocols for pulling registries.  So instead of just pulling down the JSON, he also pulled down the image.  And he also figured out he could use Scopio to push images.  And Scopio basically slowly evolved into this really cool tool that you can start to move  images around the environment.  And you don't have to be rude to do this.  So you can actually copy off of one registry, copy to another registry without ever having  pulled the image bundle to your host.  So it really evolved into a cool tool.  And we were actually working with CoreOS before we acquired them.  And we were trying to convince them to use Scopio to move images in and out of rocket.  And they said, well, we don't really want to be exerting a tool.  Why don't you make it into a library?  So we created a containers image.  So containers image is a library that now is working independently.  Now people are contributing lots and lots of pull requests into to be able to move images  around.  So it's really a base level image.  The number one contributor outside of Red Hat is actually Pivotal, who's one of Red  Hat's biggest competitors in the open ship space.  But Pivotal is using basically containers image for moving images and out of, I think  they call it garden, is their source for that.  So the next thing after you pull the image to the host, you need to be able to explode  the image onto disk.  And usually in a Linux world or in a container world, we call it, we use copy and write file  systems.  Because basically an image tends to be a layered thing.  So you basically want to install a layer of an image.  Then you're going to create another type of file system or another mount point on top  of it.  You put the second layer and you put another layer on top.  And to do that, you use copy and write file systems.  So way back when we were first starting to work with Docker, Red Hat, actually Alex,  if he's here somewhere, did most of the work, the guy that's in charge of Flatpak now,  basically introduced a whole bunch of different types of copy and write file systems into  what was Docker at the time.  That was overlayFS, butterFS, device mapper.  And so what we did is we took all that code and we moved it into an independent library.  So it was independent from basically the upstream Docker project.  And then we went and started to evolve that library.  So all these things are independent.  Next thing you need to do, so you could basically define an image.  We pull the image to the host.  We store it on top of some kind of storage.  And then we need a standard mechanism means what does it mean to run a container?  Well, luckily, OCI standardized on that.  So the OCI specification was a runtime, the second specification was called a runtime  specification, which basically says, I'm going to write a JSON file.  Everybody has to understand what that JSON file looks like.  And then I need to basically launch a program that reads that JSON file and creates the  container on the system, all the container processes, sets up the C groups, security  settings, namespaces.  So that's the last part of running a container on the system.  One other thing we needed is we needed sort of a monitor.  When I'm running a container, a container can just exit.  So when a container just exits, it doesn't know it's running in a container.  It's just a process on the system.  So you need something to actually watch that process.  If the process exits, you want to grab its exit code and store it somewhere.  OK, put it on the system.  You actually want to keep open the TTYs that are connected to it because people are going  to come to you and say, hey, what's going on inside of that container?  So you need a process that sits out there and sort of monitors it.  And that's called conmon.  So if you went to the earlier talk by Antonio, he talked a little bit about conmon and how  it's used inside a cryo.  So conmon is a simple C program that basically is the parent of the container.  So it can parent to the PID1 inside the container and just sits out there running till the container  exits, catches a sig child, and then it basically stores away some last minute stuff and exits.  That means that any one of these container engines that we're going to be talking about,  like cryo and like podman, can go away.  They don't have to stay running on top of the container to watch what's going on.  So there's a little conmon process out there running.  Did I skip ahead?  Oh, yeah.  I just explained conmon and the CNI was up there.  Sorry about that.  Someone should have said that.  So CNI is basically a way for, it's an interface, again, introduced by CoROS, that basically  defines a network protocol for the container engines to set up networking.  So other people can come in and basically get a plug-in interface to plug in different  kinds.  We're going to be using it in the cloud, we're going to be using it inside the Kubernetes,  we're using it inside a cryo, we're going to be using it inside a podman.  So basically all different types of people, tooling can basically build a CNI plug-in  and then we can use it with these tools.  So basically we have the four building, five, six building blocks here that allow us to  experiment with different types of container engines.  And we're not talking about a container engine.  That is something that you talk to and say, pull me an image.  So I know what an image is, pulls it down to the disk, puts it on to storage, configures  the run-c specification or the OCI specification and then launches the runtime.  Saves data like what happens when the container exits and reports it back to the human.  So it's basically a human interface or the tooling interface for running containers.  So one of my problems with Docker is it has become a big fat container demon.  So it's become basically a roadblock for innovation.  Having to have a demon to basically launch process on a Linux system just seems to be  wrong.  Everybody that runs the Docker CLI thinks that the container is a child process of the client.  What's actually happening is the Docker program client is talking out to the server, talking  out to a demon, and actually the process that gets launched as PID 1 ends up being a child  or a grandchild of the Docker demon, not of your process that launches.  I'm going to show you some interesting things about that.  But what's also happened is if you have only one way of doing containers, it stops all  innovation.  If I want to do some special things, if I want to move those container images around,  I have to go to the one entity and say, may I please do this?  And they say, that's not really of interest in this upstream project.  So you get denied.  So by breaking it apart, we basically get the best of both worlds.  All different tools can use, basically can contribute to these different components, and  all of a sudden you can start to build some interesting tools on top of it.  So this talk is talking about Podman.  So Podman means does everybody here know what a pod is?  In Kubernetes world, Kubernetes launches pods.  They don't launch containers.  What a pod is, is a process on the system that has one or more containers inside of  it.  So what Kubernetes wanted is they want to launch these multiple, one or more processes  all locked together in the same namespaces and then be able to move them around the system.  People came the other night, there's been some talks that talked about these sidecar  containers.  So you might have your primary application and you might lock into it another container  and that second container is basically monitoring the first container, basically figuring out  whether or not that container is or if it's doing something on behalf of that container.  I think someone was talking the other night about basically it does all the authorization  for it, so the application, the primary application can do stuff and not have to worry about authorization.  It's just a sidecar container that does it.  So Kubernetes wanted this concept where I could run more than one container at the same time  around the environment and it just manages pods.  So when we built Podman, Podman is part of our libpod effort, which is basically a library  to build pods and so we wanted to build Podman as a tool for managing pods and containers  in the environment.  But what we didn't want to do when we built Podman was basically give you a brand new  UI or CLI.  So we wanted to basically, we started out by copying the Docker CLI.  So to run commands inside of Podman, it basically uses pretty much the exact same CLI that you  use when you run Docker.  So if you find any Docker command in the world, theoretically you should be able to just substitute  Podman for it.  So lastly, before I get to the demo, we show this is the architecture.  Again, this is the same architecture that we showed earlier for the same picture that  was shown earlier for a cryo.  So we have, when I'm running a pod, I have a series of commands.  So when I'm running Podman, it's going to go out and create this environment.  So I have a command that runs, so if I'm running a pod directly, I will have an infracontainer  that basically just holds open all of these namespaces and C groups and then I have one  or more containers and running inside of it.  So Podman can run pods, but it also can run regular containers.  So the traditional way you run containers.  So at this point, we're going to demo it.  By the way, the icon here, a group of CLs is called a pod.  So that's where we got the...  So first, we're just going to do a Podman version.  We're going to be running sudo to run it as root and of course, like any good security  engineer, I don't have sudo without password.  So here we go.  We just launched this as version 0.9 of Podman.  It's all written in go because that's what the cool kids want to write in.  And what the reason it's 090 is this was...  Well, we've been releasing Podman on a weekly basis.  So it's not 1.0 yet and the 9 stands for the month and the 1 stands for the week.  So this is the first week of September version of Podman.  And so I'm going to show you info to give you...  So it's using container storage.  Containers are stored under ViLive containers.  We basically have some additional features up above.  You probably...  I scrolled off the screen, but I'm not going to go up because I'll probably screw it up.  But it's running on top of the overlay file system.  And some neat things that we've added is actually...  This is something different than Docker.  From a security point of view, we actually mount all the devices with no dev by default.  So even the images and that's showing you that you can pass in special overlays.  And this is all stuff that's built into container storage has given us all sorts of new features  that we can take advantage of.  So I'm going to cat out a Docker file.  So we're about to...  This is a Docker file that I used when I'm about to demonstrate right now is actually  Podman running a container that has Builder, which is...  Builder is another one of our projects, which is a container.  It's going to run a Builder and build a Docker file, build based on the top of a Docker file  inside of a container without giving any privileges to it.  So here we go with the demo gods, hopefully this...  So it's going to pull down Alpine image because that's the smallest image available.  If you look at it, we're actually volume mounting in here.  I have a MyVol directory and volume mounting it in.  I'm actually using SC Linux, so we're changing the label on it.  And then I have my containers, I'm bounding that into my live containers.  So Builder inside of the container in this case is going to be writing to a host directory  on the system.  And then I'm using a VFS, so I've changed the type of the storage and there it is.  So it's now finished.  It actually built a container image with no big fat demons.  So there's no demons running on the system.  There's nothing up my sleeves or whatever.  So there we're in inside of a container, another container, inside of a lockdown container  that basically built an image.  Now that image could actually be pushed to a container registry all without any special  privileges.  So you can imagine when we talked about Cryo earlier, you can actually use tools like  Podman and Builder inside of containers that are locked down.  So you can actually run really interesting workloads inside of a Kubernetes distribution.  This kind of shows it.  So here I'm going to actually show the image that was just built.  So that's interesting.  It doesn't enjoy that.  But here we have, I pulled down an Alpine image.  It actually had a couple of layers that got installed and then basically it ended up creating  my image inside of it.  So clear that screen.  So interesting thing here is one of the things I get asked often when I'm supposedly a security  guy, when I go to customers, customers are all worried about their engineers that are  coming to them all the time and saying, I got to build this Docker thing.  I got to build this Docker thing.  And I have to get access to the Docker socket.  So the Docker socket is running its root.  And you can actually do things like Docker run dash, dash privilege, dash v slash colon slash host, Fedora,  root slash host, and boom, I have full root on the system.  It's actually worse.  Giving someone access to the Docker socket is worse than giving them sudo without a password  because there's no logging.  As soon as I screw around on your machine, I can go and blow away that container.  And as soon as I blow away that container, all the logging gets eliminated from the system.  So what they want to do is they want to be able to use the Docker CLI on the system without  requiring root.  So here we are.  I'm about to show you running podman without root.  So I'm going to pull an image.  And again, hopefully the network stays up.  Everybody that's doing a YAM update on the system, please stop right now.  So it's going to pull down an image to the system.  Do this very fast, but this network is.  Talk amongst yourselves.  Right.  So this is podman running without root.  There is no daemons out there.  So what happens when I run podman as a non-root user, it actually creates the storage.  So instead of the storage being out on ViLive containers, it actually ends up being in  home directory.local slash share slash containers slash storage, I believe.  So actually I'm going to show you that I just pulled down an image to my system.  And just for that shows you the images on the host.  So there's a lot more images on the host.  There's only one image in my home directory.  If I want to run a container on top of, so there I just ran alpine container, ran the  ls command inside of a container in my home directory.  No root needed.  So what are we doing to cause this?  So how are we actually doing this?  We're taking advantage of the user namespace, which most of you have never seen before or  at least had very little exposure to.  So we actually have a tool on the builder called builder on share.  I just want to show you, it basically puts you into a user namespace without being inside  of a container.  So I am, right now I'm logged in as de-wash on the system.  That process right now that's showing you the root on the home directory, that's de-wash.  Okay, so it basically did a swap.  So what happens in the system is this is actually a program going on here.  So if I did ID right now, you will see as far as it's concerned, I am running as root  on the system.  If I did a cat of proc UID map, you would see the mapping that's going on inside of  the user namespace.  So if you're logged on to Fedora system and probably Ubuntu has the same thing, shadow  utils now puts every user logs into the system.  It puts an entry into Etsy sub UID and Etsy sub GID that defines the mapping that is available  to this user on the system.  So my system, you see as I said my UID is 3267, it maps UID 0 to that.  And it said there was one mapping in the range.  Then it says I am going to map UID 1 to 100,000 and I am going to do it to 500,000 UIDs.  So that means in my home directory now, I can map 501,000 UIDs.  I can create UIDs from my own UID as well as 100,000, 100,000, 100,000, 100,000, 100,000,  200,000 to all the way up.  And those would be mapped to one to 500,000 inside of the container.  Pretty cool, huh?  It leads to some interesting problems, though.  I can create content in my home directory  that I can't delete after I can do it  unless I'm in the username space.  If I exit the username space back to my regular UID,  I have files in my home directory  that I can no longer delete.  So I actually just wrote a blog on this past week.  Okay, so Podman has some interesting username space.  So username space has always been this nirvana  for container isolation.  I just showed you how you can use it in a home directory,  but what would be really nice is if I could use it  on the system for separating containers.  So if I run a, you know, right now,  when I run Docker or Podman,  I'm using different SE Linux labels for each one of them.  That gives me a lot of time to do that.  So I'm going to run this in my home directory.  I'm going to run different SE Linux labels for each one of them.  That gives me isolation.  With username space, I want to be able to isolate,  basically, have this container, have this range of UIDs,  have this container, have this range of UIDs,  and therefore, if I broke out of the container,  UID 5000 would not be able to interact with UID 6000  on the system, right, just like normal UID separation.  The problem is the reason we've never used this,  using the name space predateStalker.  But up to this point, there's never been a file system support for this.  So username space has been really, really cool,  but no one's ever used it.  So what we did with Podman is actually we built,  since the file system doesn't support it,  what we've done is we're using,  we're actually choning files underneath the covers  to be able to make username space work.  So here I'm going to create a username space for UID 0 to,  well, starting at 100,000 to 500,000.  That just created one container.  Now I'm going to actually look at it.  So one of the other things we've done with,  actually, guys from Suzy did this, this work,  when I run a container,  the Docker had this command called DockerTop  to show you the process is running inside the container.  So we've actually used this new library called PSGo  that actually can do something really cool.  We can actually show you the UID inside of the container,  as well as the UID outside of the container.  And this is a brand new tool, a brand new enhancement  to Podman that basically allows you to see  that inside the container it says I'm user root,  but outside of the container,  if I actually looked at that process,  the pit on that process is running as 100,000.  Matter of fact, I'll show you that right now.  So here we see the sleep program  that I launched inside of the container  is actually running as 100,000.  So now I'm going to run another container,  and this time, instead of using 100,000 for the first process,  I'm going to use 200,000 for the first process.  So now I have the container,  and if I look at that container,  I'd see that I'm running as root inside of the container,  I'm running as root outside of the container,  and if I look in the system,  I will see that one sleep is running as 100,000,  and the other one is running as 200,000.  But from inside of the container, they both think they're root.  What's happening on the file system underneath,  we've taken the Alpine file system,  we're actually churning it on the fly,  and we have some really interesting tools  that we're adding to make that churning faster and better,  all different.  So I talked earlier about Docker being this client-server model  where you exec a program on your file system,  and it talks to a server,  and that causes lots of issues.  It causes things that we can't do like things like SD Notify.  Everybody knows what SystemD does with SD Notify.  Basically, you're going to run a process inside of a container,  and it's going to call back to SystemD  and say, I'm ready to receive requests.  Well, that never worked inside of Docker  because you can't get the SD Notify.  You put the Docker command inside of a SystemD Unifile,  it's talking to the Docker daemon away on a different socket,  and the process is saying, I'm ready to Docker.  It's not saying it back to the Docker client,  it's saying back to the Docker server.  So what I'm going to show you is Podman  actually does exactly what you think it does.  So the way I do that is,  does anybody know what this login UID is?  When you log onto a Linux system,  what happens is there's a UID,  part of your process system,  records that you were Dan Walsh, right,  that you were logged into the 3267.  So this will show this on the system.  It basically says, I logged into 3267.  I can't change that. I can become root.  I can do anything on the system,  sudo anything else. This login UID tracks me.  There's no way for me to get,  change that login UID once it's set.  That means that the auditing subsystem  can record the fact that I did something,  whether I was root, a different user, or anybody else,  but it was Dan Walsh that did it.  So here we're going to run a container.  So we're running a container,  and the container is basically showing you,  inside of the container process right now,  it's running as login UID Dan Walsh, right?  It's running as 3267.  If I run Docker,  it shows that it's running as UID,  that happens to be UID minus one.  Okay? Because that's an init system,  Docker, daemon never logged onto the system.  So if I do something evil on the system,  it comes back, via Docker,  and it comes back and says,  Docker did something evil.  If I do it through Podman,  it's going to come back and say,  Dan Walsh did something evil.  So what I'm going to do here is I'm putting a watch on Etsy Shadow,  which hopefully worked.  You didn't even know it showed that?  Oh, it says the rule already existed, which is good,  because I ran this earlier.  And now inside of Podman, I am touching,  I am trying to rewrite Etsy Shadow, right?  And if I do this down here,  I got trapped.  So it shows that AUID, D Walsh, did something to Etsy Shadow.  Now if I do it through Docker,  and it shows AUID on set did something to Etsy Shadow.  So what that's demonstrating from a security point of view,  Podman executing a command basically tracks  what the user did on the system, as opposed to Docker.  And this is why I say when you give people access to the Docker socket,  it's more powerful than sudo,  because if I go through sudo and I do something in your system,  the audit system knows that Dan Walsh did it.  So it showed a little bit of Podman top features.  I'm going to run a container.  I can see the SC Linux label via Podman top.  I can see this is something that no one sees when they run Docker.  No one has any idea.  These are the Linux capabilities that are on by default  when I run a Podman.  So I've shown people are always asking, what capability?  Everybody talks about capabilities. What should I drop? What should I add?  Well, these are the default lists that you get in almost every container  you run in the environment.  If you run TRIO, we run with a lot less.  A lot of these capabilities are there just to be able to build containers.  Okay, make node.  That's so I can create device nodes.  If I'm running containers underneath TRIO in production,  I'm not expecting people to build device nodes.  So we take it away by default. You don't get make node.  We don't get rid of it when we're running it.  But basically, there's different...  If you look at running applications, there's different ways of running applications.  In this case, we need more privileges because we don't know what you're going to do with Podman.  So we call a Podman for a reason.  One of the things we want to do is be able to manage pods.  What I've shown you so far is basically all the CLI that matches up with Docker.  But here we have pods.  So if I want to create a pod, I'm going to use Podman pod.  I'm going to name the pod.  And it's going to create a pod in the system.  That actually...  The echo line there is wrong.  I didn't fix that, but don't worry about it.  I'll make this all available and hopefully clean it up.  So Podman, now I'm going to actually create another pod.  I'm going to create a sleep pod.  But this time, I'm going to tell it...  I'm going to create a container, but I'm going to put the container inside of the pod.  Okay, so this is basically creating a pod.  So I created a pod.  Now I'm assigning a container to the pod.  I'm going to create another container and assign it to the pod.  And guess what? I'm going to show on the system.  Okay, these are all the containers I ran earlier.  But basically, right now, it would have worked a lot better.  I should have killed all the containers.  But basically, I am about to start the pod.  The two containers I just created did not create containers.  So at that point, now you should see two more containers.  So you've seen containers that just got second, got created.  They were created on the system.  But basically, they just status, they got up one second ago.  So you can see, when I started the pod now, I started both of those containers.  If I want to stop the pod...  Now this is actually a bug.  When I'm stopping a container, it waits 10 seconds for the container to actually exit.  If the tool catches that and sleeps actually catching it,  there's a bug in Podman right now that it actually waits on the first one  before it sends the signals to the second one.  So we actually have to fix that so it sends the signals to all of them.  So it's going to wait about 20 seconds.  So there, we did it, and we should be back to three containers running on the system.  And again, it would be better if I killed all the containers before I ran it.  But basically, it's running.  So that shows you pods running on the system.  If I want to remove all the pods, I can actually force it to remove all the containers  that are created for the pod.  And now if I list out the pods, I'm back to zero pods running on the system.  So that is a real quick demonstration of some of the features that you can do with Podman.  So let's talk about other things.  I mentioned some of this stuff earlier, since a lot of people here are interested in SystemD.  Over the years, I came to SystemD a couple of years ago,  and I gave a talk about the CTO of Docker versus Leonard,  and both of them were being very hard on working together.  So one of the things I did when we built Podman is we wanted to have proper SystemD integration into it.  So proper, you can just run a container that has SystemD,  and it will just work under Podman without any modifications.  We support, as I talked about with login UID and demonstrated, we actually can do proper SD notify.  So we run a container, the container does SD notify, that triggers all the way up into Podman.  Podman reports a SystemD that the container is up and running and ready to receive actions.  We also do socket activation, so you can actually put Podman into the system  and have SystemD automatically fire it up,  and it will pass down the socket down to the process that's running on the system.  So all the things we wanted to do with Podman is we wanted to have some kind of,  right now it's written in Go, we wanted to have an interface to it that people could use other than Go,  so we added a remote API for it.  So we wanted to basically say, well, I can run Podman, I can set up a SystemD socket-activated Podman instance,  and then have an API that talks to it.  So we actually decided to use Violink.  So Violink is an API communication tool, a library that we can use to actually do socket activation  and communicate to a Podman running on the system, and it allows you to socket activation of Podman.  And here's a unit file that we provide, so if you install Podman, you will get a Podman,  it's actually IO Podman service file, create a socket file and a service file.  If you enable those, then you can start to run remote commands with Podman.  So here we have a Python, we also provided a Python library that you can build Python programs to communicate with containers.  So you can do an input, this is showing importing a Podman and it's listing, I think it's dumping information.  So this is dumping the information about Podman on the host, but basically it's a full API that you can communicate with Python,  via Python to it.  We also have, so we built a program called PyPodman that does all of the Podman commands,  but it's already in a Python, and the Python command talks to the Violink, through Violink to the server,  and so you can do all of the Python.  Why do we write Python command to do this?  Because we wanted to do Python on different operating systems.  So here is a demonstration of using PyPodman.  Hopefully this, you're not going to be able to see it, but...  Basically it's going to run all the commands on the system, so it's basically doing PyPodman there to list the containers.  It's showing you some of the config information, showing info, that's the Podman info command that I wrote.  So this is doing it all, and rather than, since most of you people can't see it,  the last step of this is it actually shows this is all running on top of a Mac.  So this is running a Python script on top of a Mac that's talking to a virtual machine that's running Violink  into a Podman client sitting in a system D unit file.  So we built this protocol to be able to talk from the Mac to the server we called it SSH.  So we're taking advantage of SSH on a Mac, or on a Windows box, or a remote Linux box,  to actually talk to Violink over the system.  So all you have to do to make this work is actually use SSH, set up SSH,  to communicate between the two boxes, and it'll work.  We're adding cockpit support since we have clients, so this is for stuff.  So we're adding Podman support into cockpit.  Sadly, I have it running on here, and it didn't show any images, so we're not going to show you that.  But basically, I have someone, Martin Pitt's been working to help us get this working.  But basically, we're getting full integration between cockpit and Podman,  and in this case, we're using Node.js to actually talk to the Violink protocol  to Podman running in a system D unit file.  Now, I say no big fat demons, I'm kind of cheating a little bit.  But the thing here is that Podman is just running for the connection that happens.  So we're firing up multiple Podmans for every single connection,  so we're not doing any kind of monitoring of multiple containers or anything like that.  So what don't we do? What we don't do?  Right now, so I've talked about us replacing Docker,  but the stuff that the Docker demon did that we're not doing inside of Podman.  So one of the things we don't do is we don't do autorestart,  because there's this thing called system D that does a real good job of autorestart.  So if you want a container that's going to be restarted if it fails, it blows up,  you put Podman inside of your unit file, and it just works.  We don't do Docker swarm.  We work on Kubernetes.  If you want to orchestrate lots and lots of containers, use cryo on top of Kubernetes.  We don't do notary right now, but we'd be very willing,  if someone wants to open up patches to make Podman work with notary,  we'd be very willing to do it.  Red Hat's probably not going to put the engineering into doing that.  We don't do health checks yet.  Health checks could probably be done as a sidecar container,  or they could probably be done as a system D unit file, or like a unit file we create.  Basically, a health check is supposed to come up periodically  and make sure that the container is running properly on the system,  but we haven't quite figured out how we're going to do about that.  We don't do the Docker API.  So if you have tools that talk the Docker API to the Docker demon,  we don't have a tool for that.  Lastly, we don't do Docker volumes yet.  So Docker volumes, we do most of what you think of regular volumes,  file system volumes, but there is people that have built Docker,  demons for the Docker demon that do volumes.  We're looking at doing that very soon.  So that's planned on the roadmap to be able to support that.  This point, this is the real point where we ask questions.  Anybody have questions?  Yes.  A lot of people have questions.  We're just going to hang around.  So first thing in terms of the implementation of Docker API,  there is one reason for doing that.  Maybe not in Podman,  but we can just provide some wrapper on top of the var link API.  Because without that, I don't have integration in all of the IDEs,  IntelliJ, whatever.  And without that, I am unable to get rid of the Docker from the developers' workstation.  So you're talking about twist lock and stuff like that?  No, no, no.  I mean the idea from the IntelliJ, or PyCharm, or whatever,  or Visual Studio Code, every IDE for the developer has integration.  Is all talking from a root process, launching wrapper root process.  We'd be willing to accept, if people wanted to build something like that on it,  it probably would not be hugely complicated.  I understand. It's not the scope of the Podman, but it's just an idea of wrapper.  That's one thing.  We get rid of the Docker on almost all places, except one Docker file.  So Docker file is, we support the Docker file format.  I'll sign up for a lightning talk later, and I'll talk about Builder.  So I can do a 5-minute on Builder.  Right now, Podman only supports Docker file format. Builder supports anything you want.  For the Builder, I perfectly know that I can just write simple shell script  and create whatever I want. But not everyone should write shell scripts.  The goal of Builder is to basically allow you to build another tool,  like Ansible Container, or Source to Image,  and other people to build tools that don't have to generate a Docker file in order to build an image.  But it would be nice to have one standard.  Let's let other people question that. I'll talk to you forever outside.  Okay.  Just yell.  Does Podman support a union of file systems like OverlayFS?  Yes, it runs until right now. It supports OverlayFS.  We have a bug on DeviceMapper. So it supports OverlayFS by default.  We have a bug in DeviceMapper right now.  DeviceMapper was built under the concept that only one process would do it.  But we have ideas of how we're going to fix it.  It runs on top of ButterFS, and I don't know if anybody's tested it.  We also have VFS, so VFS layer.  Someone's built an Overlay file system for...  So when you're running in a non-root environment, it's actually using VFS,  because you can't use OverlayFS on that.  But one of my engineers wrote a thing called FuseOverlayFS  that actually works real well for non-root.  So we're actually introducing that, too.  Anybody else?  Yeah.  You got to wait.  Oh, sorry.  One, two? Okay.  One of the reasons I've still been using D mostly for decompose?  Yeah, so Docker compose. I didn't put that one up.  Seems like you could...  We've been asked about that, and again, I'd be willing to accept it.  We're thinking of...  That's on the roadmap to do some kind of compose,  whether or not we support the Docker compose language,  or we support a Kubernetes compose language,  or we support an OpenShift compose language.  Again, Partman's a fully open-source project,  so anybody that wants to contribute is welcome to contribute,  and we will take patches for that,  but right now we don't have compose yet.  Anybody else?  Oh, up here.  I mean, it's actually plus one for the API,  because Ansible runs via API.  Then the word service API.  So without that...  So Ansible? Yeah.  Well, I mean, you mean Ansible container or something?  Ansible container.  Yeah, Ansible container, they're looking into moving directly to Builda  for support for that.  So to basically get out of the Docker team.  Right now, in my opinion, everybody's basically put the Docker socket  all over the place.  The Docker socket is one of the most dangerous things you can do.  As soon as you give the Docker socket,  you give full root to your system without having any tracking,  and most of the time you're doing that to do Docker builds.  Right?  So usually, you're basically giving your people the ability to create a Docker file  and then build it into an image.  Well, I just showed you that you can run...  In fact, you can run Partman non-root with Builda inside of a container  and actually build the Docker image and push it to Docker I.O.  I just said Docker image, so I owe money.  Okay?  That container image.  I blew it. I think that's the first time I really screwed up.  Okay? Anybody else?  All right.  I'm out of time anyways.  I've been...  Thank you.
Replacing_Docker_with_Podman_YkBk52MGV0Y_0_155.png Replacing_Docker_with_Podman_YkBk52MGV0Y_156_1847.png Replacing_Docker_with_Podman_YkBk52MGV0Y_1848_1895.png Replacing_Docker_with_Podman_YkBk52MGV0Y_1896_2003.png Replacing_Docker_with_Podman_YkBk52MGV0Y_2004_2099.png Replacing_Docker_with_Podman_YkBk52MGV0Y_2100_2255.png Replacing_Docker_with_Podman_YkBk52MGV0Y_2256_3047.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3048_3335.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3336_3515.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3516_3575.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3576_3719.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3720_3767.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3768_3839.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3840_3887.png Replacing_Docker_with_Podman_YkBk52MGV0Y_3888_4079.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4080_4127.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4128_4199.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4200_4607.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4608_4691.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4692_4787.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4788_4871.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4872_4943.png Replacing_Docker_with_Podman_YkBk52MGV0Y_4944_5219.png Replacing_Docker_with_Podman_YkBk52MGV0Y_5220_18215.png Replacing_Docker_with_Podman_YkBk52MGV0Y_18216_19931.png Replacing_Docker_with_Podman_YkBk52MGV0Y_19932_20123.png Replacing_Docker_with_Podman_YkBk52MGV0Y_20124_22871.png Replacing_Docker_with_Podman_YkBk52MGV0Y_22872_23855.png Replacing_Docker_with_Podman_YkBk52MGV0Y_23856_24179.png Replacing_Docker_with_Podman_YkBk52MGV0Y_24180_30023.png Replacing_Docker_with_Podman_YkBk52MGV0Y_30024_33335.png Replacing_Docker_with_Podman_YkBk52MGV0Y_33336_33911.png Replacing_Docker_with_Podman_YkBk52MGV0Y_33912_37511.png Replacing_Docker_with_Podman_YkBk52MGV0Y_37512_43151.png Replacing_Docker_with_Podman_YkBk52MGV0Y_43152_48167.png Replacing_Docker_with_Podman_YkBk52MGV0Y_48168_51623.png Replacing_Docker_with_Podman_YkBk52MGV0Y_51624_51731.png Replacing_Docker_with_Podman_YkBk52MGV0Y_51732_53387.png Replacing_Docker_with_Podman_YkBk52MGV0Y_53388_54767.png Replacing_Docker_with_Podman_YkBk52MGV0Y_54768_55439.png Replacing_Docker_with_Podman_YkBk52MGV0Y_55440_56063.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56064_56111.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56112_56243.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56244_56387.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56388_56435.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56436_56483.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56484_56663.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56664_56807.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56808_56951.png Replacing_Docker_with_Podman_YkBk52MGV0Y_56952_57083.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57084_57131.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57132_57191.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57192_57275.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57276_57431.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57432_57551.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57552_57623.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57624_57743.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57744_57899.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57900_57959.png Replacing_Docker_with_Podman_YkBk52MGV0Y_57960_58055.png Replacing_Docker_with_Podman_YkBk52MGV0Y_58056_58163.png Replacing_Docker_with_Podman_YkBk52MGV0Y_58164_58235.png Replacing_Docker_with_Podman_YkBk52MGV0Y_58236_71099.png Replacing_Docker_with_Podman_YkBk52MGV0Y_71100_71147.png Replacing_Docker_with_Podman_YkBk52MGV0Y_71148_71207.png Replacing_Docker_with_Podman_YkBk52MGV0Y_71208_71267.png Replacing_Docker_with_Podman_YkBk52MGV0Y_71268_71291.png