Packet Hacking Village - Attacking and Defending Kubernetes
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Beale, Jay
Many companies have deployed Kubernetes, but few infosec folks have experience attacking it. We aim to address that shortage, culminating in an audience-directed Choose Your Own Adventure, movie-themed demo against an intentionally-vulnerable cluster named Bust-a-Kube. You'll see how to attack Kubernetes clusters and learn what hardening techniques and freely available tools can break those attacks. We'll review the components of a Kubernetes cluster, then show how a threat actor can chain configuration vulnerabilities to pivot and escalate privilege, pilfer data and take over clusters. You will also gain exposure to a new open source Kubernetes attack tool called Peirates. Jay Beale (Twitter: @jaybeale) works on Kubernetes and cloud native security, as a professional threat actor, a Kubernetes Contributor and as a member of the Kubernetes Security Audit working group. He's the architect and a developer on the Peirates attack tool for Kubernetes. In the past, Jay created two tools used by hundreds of thousands of individuals, companies and governments, Bastille Linux and the Center for Internet Security's first Linux/UNIX scoring tool. He has led training classes on Linux security and Kuberntes at the Black Hat, CanSecWest, RSA, and IDG conferences, as well as in private corporate training, since 2000. As an author, series editor and speaker, Jay has contributed to nine books and two columns and given over one hundred public talks. He is CTO of the information security consulting company InGuardians.
 So now, this is an interesting timing for a talk because I want to introduce an old friend of ours who is, this is the first time we're going to have a talk, but then immediately you're going to have to hop over to the next room for a workshop. So this is like the prelude to your workshop, isn't it? Without much ado, JBL everyone. Hi, hi. Hi, hi. I'm going to try really hard to stay within range of the mic. And let's see how I can do it then. I'm not really good at staying in one place. I kind of like to walk around. If you've seen me at many DEF CONs in the past, I like to dance. So anyway, with that said, I'm going to talk to you about attacking and defending Kubernetes. It's a topic that's become incredibly important to me and dear to me and interesting to me. So I have a bio slide, but we all know bio slides are just big Cs of text. So I have a graphical bio slide, which basically says, this is stuff I've done or stuff I'm interested in. I really enjoy this stuff kind of going back way back to the back in the days, writing its security focus and doing CIS work and giving some early, early DEF CON talks. And nowadays teaching a class at Black Hat and enjoying a heck of a lot of work at Engardians as a CTO who likes to still be a very technical CTO. So anyway, so let's get on with what you came for. What are we going to see today? We're going to talk about Kubernetes. I'm going to do a bunch of demos. We'll do a little bit of Choose Your Own Adventure. If the tech works, the Choose Your Own Adventure part is entirely new. So the nice thing is cutting edge fresh. The downside is, uh-oh. So hopefully this goes really well. So I'm going to talk about what Kubernetes is. We're going to talk about how you, um, what you attack and how you attack it and then a little bit about how you defend. We're going to do some demos. Let's just start. So the first thing I do anytime I talk about Kubernetes is talk about what the heck this thing is for. And because, and I want to make sure I'm coming through on the mic. I might come in through on the mic and like the back of the room. Perfect. Okay. So what the heck this thing is for? And, um, and you know, people are like, I know that this is hot. I know that I should use it. That's kind of the same way we all felt about Docker. So, you know, why, why do we use this or what's made it so big? And the, basically the, the Kubernetes came out of, in a lot of ways, it came out of what Google is doing with the Borg, right? That's, it's an open source project that initially got released by Google and now has many, many, many contributors. Um, but Kubernetes basically came out of this need. We know that Google, we know that Google had just a, has a ton of computers and a ton of workloads to be clear. You know, it's not exactly the most transparent company all the time, but they've had, they have some public statements they've made. One is back in 2014, um, they were launching 2 billion containers per week. If that were evenly distributed, it'd be 3300 containers per second. Um, what's a container? It's like a workload. It's a program that's going to run for a little while and do something. We don't know how long that while is. But if you're launching 3300 a second, you can't have a human being decide which virtual machine or which host, you know, each program, each workload's going to go on. So, um, you end up needing a tool for that. Now, what else? Um, well, they did this. They were, they were launching all of those workloads onto 2.5 million servers. At least that was their number back in 2016. And in 2016, hard drive failure rates were just right below 2%. So if every server had just one drive, and that's a big, you know, hand waving thing, but every, if every server had one drive, that meant they had 133 drive failures a day or one every nine minutes. So now you've got, now you've got a resiliency problem, right? Like you wanted to guarantee, you know, you wanted to guarantee so many nines of uptime. And yet your physical machines are going to, you know, entire physical machines are going to die on some regular basis. Like say, read notwithstanding, one every nine minutes. So what features would you want if you wanted to be able to stage a ton of workloads over a ton of servers, figure out how to use the servers as the greatest, as close to their capacity as you could. So you weren't buying lots of extra machines. And you wanted to make sure that as programs crashed and as hard drives failed and so on, that you still had availability. So you end up with what Kubernetes is, which is basically a software defined data center. And we do that by container orchestration. So what do you get? So basically this is your feature set. This is what you end up saying. You end up saying, well, sometimes I have a lot of people checking their email on Gmail or searching right now for the thing that just happened in the news. And sometimes I don't have as many. So I'd like a whole lot of scaling. I'd like that horizontal scaling. I'd like to be able to say I have four copies of that workload. No weight. I have 400. No weight. I have 50 and so on. And you get automatic bin packing, which is that like which workloads should go on, which machines. We can keep all the machines as full as we can. And you get self healing like that hard drive crash thing. And you get a whole bunch of other cool features. You get like automated rollouts and rollbacks where you can basically deploy multiple versions at once and see if one of them gets, see if one of them ends up producing more sales or more of whatever it is you're trying to produce. Or see if one's buggy and then be able to basically route all the traffic to the one that isn't and so on. You get other kinds of things you would need if you're going to do microservices. So you get service discovery and you get load balancing. And then you get orchestration of your storage and you get secrets and configuration management. So that's kind of, that's what the purpose was. I'm going to do a little bit more theory. So just kind of like what's a pod? What's a container? So right, we, when we're talking about Kubernetes, we're talking about containers, but we say pods a lot. And basically pods are the smallest unit of computing Kubernetes. So all the containers in a pod are sharing an IP address. The only reason you put multiple containers in a pod is if you want them to absolutely positively land on the same host. Like they have, they have need for very, very quick network access or what have you. Maybe they need the same local storage. An odd thing about pods is all the containers in a pod share an IP address. Okay. So then you've got Kubernetes nodes. Simply put a node is any system you're going to run Kubernetes on. A node is any of the, any of the systems in your cluster, whether those are virtual machines or physical hardware. The nodes run a, well, first and foremost, the nodes to be a node, you've got to, you have to have a container run time. So you have to have something like Docker run, see, or what have you. Something that's going to basically take instructions about, take instructions from Kubernetes about creating containers and create them and maybe destroy them and so on. You also need a kubelet and a kubelet, Kubernetes kubelet. A kubelet is just this thing that runs on every Kubernetes node and it ties the, it ties the container run time or the node into the rest of Kubernetes. It talks to the API servers and so on. Then you've got a kube proxy and that thing manipulates IP tables rules to create cool networking. It also serves as an actual proxy. So, yeah, we'll get into that a little bit more. And then services. Services are big because in Kubernetes and services give me a great occasion to kind of describe how you talk about the names of things in Kubernetes. When we're doing the attack demos, we're going to be talking about this pod and this pod and this pod. But for the most part in Kubernetes, you don't ever think about which pod names you want, right? It's, the whole idea is that the individual pods, right, the containers, they're very, very much supposed to be cattle. They're not supposed to have names. They're supposed to be interchangeable. I'm supposed to be able to trash them, start new ones up. I have to, I'm going for, I'm going for that. So what does that mean? I mean, if I want to talk to a pod, what do I do? You don't. I mean, you might, but rarely, mostly you talk to a service and a service is mapped to multiple pods. Think of it as an internal load balancer. Now, now, if you've got, like, why am I, why do I say that would get me into something cool? That gets me into labels. And basically, you have a load balancer. You can't say this load balancer should be pointed at these five pod names, right? First, you're going to have a whole lot more. They're going to get added. Second, those pods are going to die at any moment. Third, you're trying to go really, really fast. Remember, you know, not 3,300 containers per second being launched, right? So what do you do? You end up basically labeling everything. So you label all the pods. You give them labels, and those labels are basically, sometimes they're by convention, sometimes they're whatever you chose, but you label the pods and then you create a service. And you say everything that, you know, this service is going to be a name, an IP address, a port, and everything that goes to that name and port or IP address and port is going to be sent to one of the pods that have the set of labels the service specifies. That does kind of open an interesting question. Like, if you are allowed to stage pods onto a cluster, could you start intercepting other people's traffic by just giving yourself the same labels that their services are looking to land things on? Okay, interesting. Last thing we talk about, last big, big, you know, kind of simple glossary concept we need is namespaces. The namespaces are just a logical grouping for things. So we'll see those. They're often used, they're often used to separate projects or to separate teams, to separate an individual, you know, maybe to separate, well, anything that you, anything you choose. So there's some way of chopping things up. And it's important to say Kubernetes namespace is not the same as a Docker or Linux namespace, so much so that when I teach classes on this, one of the other folks in Guardian, Jared Fredes, asked if we could please call Kubernetes namespaces, space names, since Docker and Linux, you know, already have namespaces. So it was, you know, don't overload the term. So here's a quick glossary slide. I'm going to say a tiny bit more on theory and then we'll have some fun. So, so Kubernetes is basically works in this declarative model. You don't generally say, you're going to see me generally like, hey, do this thing right now. But you don't generally tell Kubernetes to do a thing right now. You don't say, hey, take this, you know, take this container image, start a container on that note over there with it. Right. You tend to say, you know, it's very, it's very delegating and hands off, right. So the idea is you say, Kubernetes, I have this program. I've created an image, maybe I've placed that image on Docker hub, maybe put it on some other container registry. And I'd like you to now like download that, like have some nodes, download that image. I'd like at least say five to be running it. Like if you need more scale up, but you know, I want five, no less than three, like let's, you know, just do me a favor and get that running. And Kubernetes says, sure, and it's got a bunch of like control loops that basically first make that the state of the world. So you describe the state of the world you want it, it basically spends its whole life trying to keep the state of the world where you want it, where you've said it has to be. And then it basically loops and it says, ah, a machine fell down. I guess I'll have to reassign those workloads or looks like something's not responding. I'll have to, you know, I'll have to, maybe I'll kill that off and I'll start something else. So you describe the state of the world. I think this is basically my last theory and I hope I haven't given you too much. So the kind of target components, there are a bunch of components you'll target. The big ones that you hear about are especially on the first two slides here. So you'll target the API server, the thing that everybody's basically doing all of these HTTPS and God forbid HTTP requests against and asking to do things. And the cubelet, which we said bridged the, bridged each node into the system. The container runtime, you can often think of Docker and I'll often say Docker, but there are many, there are multiple container run times. And that container run time is the thing that pulls the container images from the registry, you know, like Docker Hub or what have you, and instructs the kernel to start up a container. You have an, you have an LCD server that maintains all the state. And if you ever get access to that thing, yeah, there's not really a lot of chopped up authorization. So if you get access to it, you kind of own all the, you own all of the persistent idea of what the system is. And so that's very good for you. A Kubernetes dashboard, that was the thing that got a, as I recall, that was the thing that got Tesla in trouble where their dashboard was available to the internet. And I think was allowing anonymous access to do anything. But that's a web interface as I, I think it's being deprecated. There's metrics components and different kinds of cluts, cluster services that'll give you good information. Those, those are your usual targets. Just to throw in two more, two more components that people that it's really helpful just, you know, if you're operating Kubernetes, defending it or attacking it are useful to know. One is the controller manager and the controller manager, it makes it sound like it's like the controller manager would have lots of separate controller programs. And it kind of does, but they're all just compiled into one big, you know, just compiled into one big program. So the controller manager has all of the controllers, the controllers of things that basically do those loops and say, is everything the way it was supposed to be? Is everything the way it was supposed to be? Ah, something's different. I'm supposed to have five copies of this. I've got four. I'll go talk to the next one, the scheduler and say, excuse me, scheduler, can you find somewhere to, can you find somewhere to put this? Can you find a node we can put things on? We tend to target these components only through the API server. So attacks on Kubernetes, this is where we start to finally get to attack and defense. Attacks on Kubernetes, they start from, they start from the perspective of compromise pod. So you can attack, you can attack a Kubernetes cluster from across the network and it's even less effective than attacking most companies, you know, across the internet. Across the internet has been, right, lately. We've gotten pretty good at parameters. Kubernetes is pretty darn good at parameters because for the most part, there's no traffic being allowed in unless it's had some kind of ingress defined for it. So for the most part, you attack a Kubernetes cluster from the perspective that you are in a workload, that you're in a container and that container is compromised or otherwise under your control. So what are the ways the attacker might end up with that kind of control? One is, yeah, as you'll see in our demos, this is the vector I like to show. It's the attacker found a vulnerable thing that you were hosting in Kubernetes, got remote code execution and now they're in a container. And from that container, they may have the ability to do much more far-ranging things because we're in an orchestration system, a very automatic one. They may be able to do far more wide-ranging things than if they just busted that, you know, than if they just busted a, you know, some web server or something that was in a cloud environment or in a data or in an on-prem data center. So how else would the attacker have gotten in? The attacker might have just fished somebody. Brad Giesman, who I'll, whose name I probably just butchered and whose Twitter handle I'll have at the end of these slides, has a great slide in one of his presentations about how Kube control apply and that a URL is the new curl URL into shell, right? If you, one of the great ways to fish somebody would be if you put up a how-to article on Stack Overflow or what have you, and the, you know, part of that article was, oh, just apply my, you know, apply my YAML from this URL. You can get anyone to apply the YAML from the URL, then you've got places for your crypto mining to go. So, and our attacker could just be some internal threat actor who's looking to escalate their privileges. I often, you'll, you'll often hear me use the word user and attacker interchangeably on the defense side. That's not really, because I think all my users are hostile. I just don't know which ones of them might be. I don't know who's being coerced. I don't know who's, you know, getting curious. I don't know who's actually got an axe to grind. And I think we all know these privileges are a good idea at this point. So, okay, so, so if an attacker is going to be in a pod, what are they going to do with that? I'm going to give you a few, I'm going to give you a few things there's even more, I'm sure. So, this is kind of the general way. So they could use just the network access that they get provided by the pod to access other services. And I'll say very, very, very often, especially in the microservice model, a Kubernetes cluster has a bunch of things that are all supposed to talk to each other. And it's actually, it can be a little difficult or at least take some work to sit down and say, okay, what should and what shouldn't. So once you're in the cluster, you have access to a whole bunch of things, to a whole bunch of things, many of which might not be intended to be communicated with by anyone on the internet or anyone in the, anyone at the service that you've pond is trying to serve. So what else? The attacker could attack other containers that are in their same pod. Multiple containers in a pod, they share, well, I told you, they share an IP address, they share a local host. So it's a cool little Linux namespace thing, which we won't get into unless somebody asks me later on. What else? So the attacker could make requests to the API server or to a kubelet and ask those to maybe run commands in another pod, maybe even interactively, to start a new pod that has privilege. And that privilege could be all kinds. It could be a privileged pod, which means it'll start on the host and have access to system devices and be able to, among other things, maybe mount the hard drive, a little of the node. But there's all kinds of privilege you can get out of starting new pods, and I'll show you that. They might gather secrets that Kubernetes provides to pods because Kubernetes has a whole thing where it needs to provide configurations to pods, because you're really supposed to make those pods come up with a static image. So the idea is the pod starts with a static, totally replaceable image and pulls whatever information it needs from elsewhere. You might connect to the Kubernetes dashboard to perform actions. Maybe it wasn't accessible from outside the cluster. God, God hope. Or gosh, hope. What else? The attacker might find a way to interact with the LCD server and change the cluster state. It's a nice way if there are network access control rules, if there are network policies in place that are getting in your way. If you can talk to the thing that stores them, yeah, you could potentially change them. And the other one, the one that's really fun to me for anything that's either in a cloud provider or even in a non-prem cloud, is that the attacker might interact with the cloud service provider, because many workloads will one way or another, and I'll show this to you too, will have privilege on the cloud provider. So I'm going to show you a bunch of demos, and these demos are real. They're often things that we do in classes I teach. They're also, I'm going to tell you later on, we have an open source project, and it's an intentionally vulnerable Kubernetes cluster. Call it like damn vulnerable Kubernetes cluster. This one's called Bustacube. We've got an Easter egg Bustamoo video that you can find if you go to the bustacube.com website. But so some of the things I'm showing you are basically available as scenarios that you can play in this open source free damn vulnerable Kubernetes cluster. So I've done a thing where I've made my videos part of the slide, and so now I'm going to see if that actually works. Please work. And if it doesn't, we'll have to use YouTube one way or another, or something else. Okay. Color me very embarrassed. Let's just see if this one's the one. You should always test your demos, even when you record them. So I tell you what, we're going to find another way to do it, and I am going to, let's see, I'm going to unplug from the projector for a second, because that means I'm going to go and look at my own file system and I'd like to not share it. And here we go. Just embed the videos in the slides, they said. It'll be fun, they said. So I'm pretty old enough to have played Warcraft and you click on the guy and he's like, what? And you click on him again and he says something else and he says, you know, join the army, he said. See the world, he said. Okay. So here's our demo. And hopefully entering, oh, well, that's not a demo. Is that a demo? That's not a demo either. Anybody seen my screen? There we go. Let's just hit this on mirror and call it a day. Okay. So that's not the right. Don't mind me, I'm clearly a noob. I may need alcohol. So this is one of the Bust-a-Cube scenarios. So you go up and you see a WordPress site and the WordPress site, I will tell you, is hosted in Kubernetes. And you fire up maybe Durbuster to go looking for hidden content and Durbuster takes a little bit of time, but we can run it and let's go a little faster. And what we find is backdoor.php. So when we surf to backdoor.php, this is what happens. I get this thing where basically this WordPress instance, like so many other WordPress instances, has been pre-compromised for your convenience. So I type ID and it says you're dub, dub, dub data. And hopefully I shut down Durbuster because it always choose a ton of resources. It's something we always say in class. It's running slow. Have you turned off Durbuster? No, it's 500 requests a second. Well, that'll do something. So I'm going to stand up a quick little web server in Cali because I found that reverse shell. So I'm going to stand up a little web server. And my purpose of the web server is to be able to hand a, to be able to hand a interpreter or a, you know, or if I wanted a, you know, whatever, some other kind of reverse shell. But I'm going to go for a interpreter. And so I just put in here like, hey, could you please pull this interpreter binary I've created from my web server and while you're at it, make it executable. And while you're at it, run it. And I like to fast forward my videos because who doesn't. And go look at the, you go look at the web server and Mrs. Bin has been pulled the interpreter binary but also a copy of Kube control. The command line utility that you use to do damage, I mean, to administer or use a Kubernetes cluster. So I'm going to pull Kube control. And the other thing I'm going to do is just take and make sure that it's landed here. So here's Mrs. Bin, my interpreter binary. Here's Kube control. Cool. And now if, if recorded J will type a little faster, we'll run the interpreter binary. And I'm going to go over to my Metasploit console and actually be like, oh wait, I have an incoming interpreter that needs to connect to the console. I should probably start the console. This ordering is just great, huh? But that's often how you work. So I'll interact with that session. And I'm now in a pod. I know it doesn't look like it. It looks like an ordinary Metasploit, you know, Poned Metasploit instance. So, but I see my copy of Kube control. I'm happy. And when I run mount, I get a tremendous number of file systems. So I'm not looking at one hard disk or what have you and a bunch of them put there by Docker and by Kubernetes. And so the one that's really interesting to me is this one that says run secrets Kubernetes service account, IO service account. As soon as I see that directory mounted, I know that I'm going to have some level of access to the API server and I'm happy. So I'm going to go and first I'm just going to go and collect my flag. I built this scenario as a capture the flag. So let's look at our little flag. And it says, hey, you've found the servers back door. You might notice this isn't your usual Linux system. You're in a cluster, my friend. It's time to bust a Kube. This container doesn't have any more flags. Can you find your way to another pod? And tint. So cool thing in Kubernetes is, like I said, one of the things you can do is see if you can move to another workload. So you're in a container. Can you move to another one? So I'm going to see what my service account can do. I'm going to go and look in that directory and that Run Secrets Kubernetes IO Service Count Directory. If recorded J will type just a little bit faster. And I look at that namespace and it says you're in the marketing namespace. So that directory has a token. It has the certificate authorities public cert. And it has a description of what namespace you're in. And that namespace, like I said, is kind of a wave organizing. So we're in the marketing namespace. We'll come back to that later. And then I'm also going to go and look at the environment variables that get pushed into every pod. And my environment variables will tell me where to find the Kubernetes API server. So there's this assumption in Kubernetes that every workload will need to ask Kubernetes to do things. I don't know if that's really the greatest assumption, but I think let's just call it not the strongest of defaults. So I've got the API servers IP address and port. And I'll clear my screen and I'm going to set up a quick little kube control where I'm saying alias kube control to the path to kube control. The token pasted in from this long path to the token, the certificate authority path, the namespace and the server that I just got. And so every time I type kube control, all that stuff will be passed in. So let's just kind of skip and let's see what did I do. So I said, let's do a quick get pods. And get pods is honestly my go to just to figure out if my setup is working. And so it says, oh, there's two pods. You're the WordPress one. And I ran host name and it says, that's the one you're in. And then the WordPress my SQL one. And that's another one. So let me see if I can move over to it because that was what the hint said. So I'll do kube control exec it looks a whole lot like Docker exec it. And what program I'd like to add to that container. I'm going to add a shell. I get this kind of error, but I'm in the container. I've just moved over and I've probably moved over to a different host. Like I'm probably somewhere else in the data center now. So there's another flag and we'll go get our flag. And I can't clear my screen because my terminal is not set. So I set it to VT 100 because that was one of my first back in the BBS days because I'm exceedingly old. And it says, hey, you found your way to another container. Actually, you're on a whole new virtual machine. We're moving from one container to another in a Kubernetes system, often mean switching nodes. Your end goal is to get to developers non-containerized host. So it's to get not just to own the cluster or potentially, but to get to a machine that's not in the cluster. So to get there, you're going to want to see if you can hop tenants. Tenants are usually separated by namespaces. So let's see what we can do. I know I'm fast-forwarding through my stuff, but so I'm in the other, I mean, that MySQL want to try to get pods. And the problem I'm running into is like, hey, you're not allowed to get pods. So the cool thing is I'm in a different service account. I was in the marketing WordPress service account. It's allowed to get pods. The marketing MySQL one can't. So over time, I might want to like go around and pick up all these service accounts and like hold on to them. Remember, it was just tokens. So if I can store all the tokens, I'll have like, I can end up building privilege wider than just what one thing had. So I did get nodes and I found out the IP address for node. Here's a name, but this node has an IP address. This is, this one is, this one's running in the cloud provider. And there's a cool thing. So you can talk to the KubeLit and this is, thank gosh, finally not a default, but some clusters will have it. You can talk to the KubeLit on, it's got a couple ports, 10, 250. I'm going to go and say, can you give me a list of running pods? And it gives me JSON because everything in Kubernetes is JSON and YAML. So I'll take that and I'll run that again. And I apologize for this big block here. So I'll run that again and I'll add some JQ, JSON query, really good tool to learn. And I'm going to say, give me all the pods, kind of better format it, give me their name and their namespace. And what I'm looking for is to see if there's a pod that's in, that's in another namespace besides the marketing one. And a quickly fine one. There's a dev pod. And that dev pod, so Kubernetes loves to tell you about anything you want and that's really, really nice as an attacker, especially if you can script it. So I've got dev sync, I've got dev web. These are two containers that are running in the same pod. And I've got the image they came from, so they're up on Docker Hub, you could pull them yourself if you wanted. And as an attacker, that's often what you're doing is basically saying, well, what workloads are running, and you can go look in container images and see if there are any secrets or anything that would be really helpful to me. It's very, very nice of the defenders to hand you descriptions of their workloads. So I'm going to look at that, I'm going to run another command, and this is, I did running pods. I'm going to tell the cooblet, hey, would you mind just executing a command for me? So I'm going to pass it run, and then the namespace, the pod, and the container name. And I'll do command equals ID. And I get an error code that says basically no, either there's no shell in there or I'm not allowed. I will tell you that there's no shell in there. So, but remember, there were two containers in that pod. So I saw dev sync and dev web, and that's a really common thing you see outside of containerized environments, like you have a web server, and you have something that's there to pull it content in from wherever that content's being created or stored. So I was hoping I could get to dev sync because it'll have that, it'll have the credentials for pulling content. It'll have an SSH key. I couldn't, so let me try dev web. So dev web, I can run commands in. I ran ID, it says, yeah, that thing's even running as root. Very nice. And so let's try a different command. Let's look at the file system. Oh, look, another flag. I like flags. I love doing CTFs, and I love doing CTFs actually like first doing them as an attacker and then going and basically doing them as a defender. It's saying how many, how much of the attack can I break with proactive stuff, with stuff that you could have thought of before you knew that you had the vulnerability. So this thing says, hey, you're not in the marketing department anymore, you're in the development department's namespace, so can you go take over the development department's non-containerized machines? And they're big fans of the matrix, so it's named the mainframe, named for the machine's mainframe that may or may not have been in the second and third movie depending on your religious beliefs. There are many people I know who acknowledge the truth as they believe it, that there were no second and third matrix movies. Only quests for more money. So here we go. So we've got, we're trying to do a get pods. So I am, I'm in one pod asking the cooblet on another host if it mines running commands in another pod, and I'm asking that pod to run coop control commands. I find out, okay, dev app isn't allowed to list pods, but it's kind of cool. I now know that I've got a third service account. I've got, I've got the dev dev app one. So I'll try get secrets. Let's see what secrets are available. Oh, nice. There's a machine, there's a mainframe login and an SSH key. Well, the two pods in there, the two containers in there are going to likely have access to the same secret. They've been placed in the same place. They don't necessarily need it in dev web, but what the hell? Let's go and ask for a copy of that secret. And I promised the next demo is hackers themed and choose your own adventure. So we're going to have some fun. So here's our SSH key and you're like, that doesn't look like an SSH key, and that's because it's a secret. It's, it's opaigly encoded. It's base 64 encoded. So we can decode it and boom, there's our SSH private key. And we can do the same thing with the other secret here. So we can find out basically what is it used for. So I'm just going to fast forward a little bit and tell you that it's the, that it's a user, it's a user and a host in AWS, but not part of the cluster. So now we could SSH into that and we're in the developer's mainframe. And it says, hey, you made it to the machine mainframe. Can you, for extra credit, can you get credentials to take over the whole cloud environment? And, and we can do that, but I'm going to set the rest of that aside because we do that in another demo. So, so let's see. Let me go back to the slides for a second. And here was our, and I'm sure that I'll be, because the vagaries of having the videos, you know, in a separate program, I'm sure I'm going to be doing a little bit of going back and forth and having the whole screen do this. So here's our attack path. We got a interpreter in the WordPress container. We moved into MySQL container with the kube control exec. We used the MySQL containers totally unfettered network access inside the cluster to reach a kubelet that we wouldn't have been able to reach from outside the cluster on the master node. And that kubelet let us basically take a look at a, take a look at, take a look at run commands in another container in another namespace. And from that other container in that other namespace, we found, we figured out there was a, there were secrets. There was an SSH key. So we got that SSH key. We authenticated to the high value developer machine and later on we ended up getting to take control of the AWS account. And we're going to skip that right now because we're going to do that in a much more fun way. So we're going to play hackers choose your own adventure. And I'm going to drink some water for a second because whoa. So I like the movie Hackers. It's a, it's a fun movie. It is an oldie but goodie. It is thoroughly realistic. And we all know that whenever we're hacking, we, we see like strange translucent 3D models and, and you know, of buildings and we fly through a city and it's really cool. And all of that was on, was clearly done with the technology that had Apple twos and floppy disks. So, I mean, I remember 3D rendering back then. I wasn't doing it. I was watching people do it. And yeah, no, not on that. So here we go. Let's start here. And, and again, this is where I have embedded video. So we will, unless I click on that and it works. Let's see. Oh, hey, look at that. Now can we full screen that? Is there a way? No. So we're going back to oldie but goodie. And here we are. First, this is going to look a little familiar. So we've got a, we've got, what's that? Oh, you're on the slide and I'm on the, and I'm on the video. Okay. Boom. And switch. Come on. I'm going to Windows computers soon. And that's a lot for me to say, trust me. So anyway, here we go. So we've got the, we've got the Gibson control room. We have Penn from Penn and Tower, who's apparently a techno weenie according to Ben from Short Circuit, who somehow was playing in Indian and talking about Mr. Johnny Five. But Mr. Johnny Five here was the plague. And we have this nice little form that's called the Gibson control room. And the Gibson control room clearly lets you like leave a message. So it's based on a, it's based on a actual, on a Kubernetes documentation vulnerable app. They didn't mean for it to be vulnerable. But it's only a little vulnerable. So I've added a vulnerability to that. So the, so what we've got is I look at the source on that page and I find that the form is basically being provided by some jobs. So if I go look at that JavaScript, what happens when you submit the form is you're, you're referencing guestbook.php. You're sending to guestbook.php. You're saying, hey, command set key messages value this. Okay, really? You're going to let me just, you're going to let me decide what, what key I'm going to set. You're not just going to have me set whatever key, you know, was meant to be set by this form. So that was their bug. And what I've added to it is I have something that actually runs code out of a different redis key in this, you know, back in redis database. So we go and take the Gibson control room. And if we were to type a message, you know, it shows up, this is the intended, this is the intended purpose. But instead, we're going to paste that same thing that happens when you submit the form, except instead of the message key, we're going to try a different key. Or we'll try a, let's see how it works. We're going to hit foo. And we'll put in just some key, some random key. Oh, cool. Okay, so JavaScript is probably updating that key. Let's try going to status PHP. So I'm asking you to imagine that we did a derbuster or looked at robots.txt. We found a status PHP. I'm not doing it because, well, I'd like to spend our time on Kubernetes. So I go to the status PHP page and says, hey, no command to run. I'm going to set the reddest key, the reddest command key to this so that you can reload the page and that will get run. So if I reload the page, I get the output of, you know, I get the output of scrolling that web server. It says all services are operational. Here at, what was it, Ellington mineral? Ellington mineral, Ellington, I think. So as long as there's a command key, let's go and set it. And I've set my command key now to environments, to run environments, like we did before to find a Kubernetes cluster to find the Kubernetes API server. And here we go. So we know that once we're in a workload, if we reach out to 10.96.01, we'll get to the Kubernetes API server. So I'm going to go a little further, kind of similar to last time. I'm going to curl in. I'm going to tell this thing, excuse me, would you just pull in a interpreter binary I created and connect back to me? So I have it pull it in. I set that key. And now when I run status, the page hangs. I love when pages hang when I've just given them binaries because it generally means the binary is running successfully. So if I go over to my terminal where I've got a menisploit handler waiting this time, I find there's a session waiting for me. And the first thing I'm going to do before I start interacting with a shell there is I'm going to upload, I'm going to upload a copy of kubectl, which I've stored on this Cali system in this directory. So I'll upload kubectl and I'm going to upload some YAML, which we'll get to later. So I've uploaded some YAML, which we may or may not get to later depending on your choices. And I've asked for a shell, had PWD. Looks like my video is cut off by one character. A hit ID, it says your www.data user. I try SSHing. Well, I don't know what I tried because that was the end of that part. And now we're going to do this part so that we're a little bit gentler. I'm going to ask you, so do you want to try the YAML files that I put up were ones that would mount a volume on the host? Like they try to mount the host's own directory, the old docker, start a container mounting the host. So do you want us to start the host mounting pod and see if we can take over a node? Or do you want us to exec into another pod? I heard the host mount first, so we'll try that. So let's see. Boom. And boom. Here we are. We're going to run that front end. And we'll set kubectl executable so we can run it. We'll go and set up our kubectl. We'll try to run kubectl apply. Realize, wait, we've got to set all those parameters we set before. Sometimes you get lucky. And so I'm going to take that same directory, that same run secrets kubernetes.io service count that's being mounted into the pod. I'll set a variable called dir to that, export dir to that, just to make this a little bit easier to type, a little more manageable. So if I look at the dir, got the ca cert, namespace token just like before. Let's go and run. alias kubectl is going to mean that, same way we did before. So servers out on 10.96.01. That was what we found from that environment variable. And now let's look at our alias. That's that token being pasted in. So kubectl get pods. Cool. We've got a front end. We've got front end pods. We've got a redis master. We've got an Apache status, that thing we saw before. We're in the front end. So let's try launching a pod. kubectl attack, so kubectl apply. I like it as kubectl attack pod. And it says, sorry, your service count front end isn't allowed to create resources. So, and this is where I get to my slide. So we'll do that real quick. I was considering for this one, oh, damn it. Let's just keep the screen mirrored. So I was considering for this one, doing a spandex quote from the movie, and I decided that wasn't a sentiment I wanted to voice. So I see that I'm going to make sure that I'm moving on time. So at this point, we'll go back to our original, we'll go back to the other thing we could do. So our two options, where do we want to launch a pod or exact in the back end? So let's exact in the back end. And, or let's try exacting into another pod. So we'll go like we did in our other demo, we'll try exacting into a pod. Keep wishing we had a better terminal. kubectl get pods, I see that there's a redis, there's a redis, that redis setup. Remember, I was pushing redis keys. I might try to investigate the redis and see what else now that I've got more direct access to it, because redis is very often, well, almost always unauthenticated. But let's go into the redis pod. So I'm rooting there, I'm redis master, I'll grab a copy of the interpreter binary I've been using, just because that's a very convenient way of putting files into something. And I'll background that, and I'm going to switch over now that I've got the redis master, that pod running my interpreter. So I'm going to switch over and catch that session. And so in that session, I'll go and upload the same kubectl that I did before. I'll also upload that attack pod that I just used before. And I'll also add something called attack demon set, just in case we need it. So, now we get another choice. So you're in the back end. Do you want to try host-mounting pods, or do you want to check out the cloud APIs? Cloud APIs? Cool. Okay, so here we are. And I'm going to show you this in two ways. And can I get a rough time check? How much time will you let me be late? Oh, less than five. Okay, so this is where I normally kick on the speed control. So, okay, so we're in a pod. We're in that redis master pod. And we say, hey, I'd like to talk to the metadata API. So every cloud provider gives you 169254, 169254. That part's common. What the calls are that you make against it, they document wonderfully, and you read the documentation. So this is on GCP. So I'm going to go and say, can you tell me what the project ID is? I need that project ID to do API calls. But then the more interesting one is usually where you say, would you mind, in addition to the project ID, which I'm going to set into a variable, would you mind telling me what service accounts are assigned to the machine I'm on, so the machine this pod's running on? And there are two, but default is mapped to that other one. So I'm just going to use default because it's easier typing. And I'm going to go and say, okay, for that service account, would you do me a favor and just give me a bearer token that I can use to act as that service account? And it says, sure. Here's your token. It expires in like a half an hour or an hour, but I've got a token. So I'll set a variable to that. And now what I'm going to do is say, let's go and I've set that token to that. And I need to refresh that token so most of my commands will pull that token over and over, every command. So I'm going to use that token, type out authorization bearer token where the token's provided, say I'll take JSON, and now I'll say, hey, for this project, I'd like a list of buckets. And it says, oh, there's only one. Okay, well, what's in that bucket? And so I basically say, give me a list of objects. And the way I do that is it's like slash B for a list of buckets, then it's slash B, that bucket, slash O for a list of objects. And then for those objects, I can run through them. Now, on this one, I don't find anything that's all that useful. I don't find what I find in some cloud environments. I'm going to show you in another demo somehow. But I do find a kubion v. So I can pull in a metadata instance for this cluster that was set by the installer. And when I do that, what I get out of it is, among other things, a couple service count tokens for the cluster. And that ends up leading, which I won't now show you, but that ends up leading to win. And so just for the end of our, for the end of that, that then leads to our crash and burn. So if Ming will let me have one more minute, I will show you a cooler version of that. We have a tool that we keep soft releasing, but never kind of like putting up on Twitter. So it's called Parattis. And Parattis will, if I actually just, I'm just nervous. So it's in here, but somehow I'm, let's do this. So Parattis is basically a remote access Trojan for Kubernetes. You run this and it does all kinds of things. And I'm going to start this from the beginning, but it is under two minutes. And then I will get the heck off the stage and tell you that you should ask me for stickers. So it's got a bunch of things. One of those I had said, okay, 13, go get the IAM credentials from GCP. The same thing we just did, get the cloud service account. And then, come on, Jay, type faster, highlight faster. So that's for that same one I just got. And now let's take that and let's pull, let's go looking in GCS in the storage buckets. For service count tokens. Oh, look, this installer, this one was cops, in left service count tokens. There for all the components, including a cluster admin, which would be perfectly useful. And so that's the like, basically, that's probably roughly what happened to Capital One recently, where it was talk to the metadata API, get your cloud account, use that to go rating for information or access. So anyway, so now once I've done that, Parities will basically store every, every count it gets. And then you basically say, okay, I'm going to switch context. And you go and you go through and you collect all of the service count tokens, and then you basically have this big old store that's your loot, and it keeps it in memory. And you run around and basically get more access and get more access. We've even got a, this video will be up on YouTube, but we've even got something that basically takes over the nodes. If it's allowed to mount a host path volume, it'll go and add a cron job that will send a netcat shell back to your, back to the IP address of your choice. So once we have this cluster admin, we're allowed to, we can go and do everything which included like, hey, now show me all the secrets. And all the secrets includes all of the tokens, all the service counts for every component. And, and Parities will not just grab, will not just list these and get them, but it'll also store them and let you use them. So it's a fun little pen test tool, and it's up on GitHub. We'd love some help, we'd love some help producing it. So it's an inguardians tool. We do a ton of Kubernetes work in inguardians, and I will now get off the stage and take questions all the way out in the hallway. So if you see me, please ask me for a sticker. Thank you.
 So now, this is an interesting timing for a talk because I want to introduce an old friend of ours who is,  this is the first time we're going to have a talk, but then immediately you're going to have to hop over to the next room for a workshop.  So this is like the prelude to your workshop, isn't it? Without much ado, JBL everyone.  Hi, hi. Hi, hi. I'm going to try really hard to stay within range of the mic.  And let's see how I can do it then. I'm not really good at staying in one place. I kind of like to walk around.  If you've seen me at many DEF CONs in the past, I like to dance.  So anyway, with that said, I'm going to talk to you about attacking and defending Kubernetes.  It's a topic that's become incredibly important to me and dear to me and interesting to me.  So I have a bio slide, but we all know bio slides are just big Cs of text.  So I have a graphical bio slide, which basically says, this is stuff I've done or stuff I'm interested in.  I really enjoy this stuff kind of going back way back to the back in the days, writing its security focus  and doing CIS work and giving some early, early DEF CON talks.  And nowadays teaching a class at Black Hat and enjoying a heck of a lot of work at Engardians as a CTO  who likes to still be a very technical CTO.  So anyway, so let's get on with what you came for.  What are we going to see today?  We're going to talk about Kubernetes.  I'm going to do a bunch of demos.  We'll do a little bit of Choose Your Own Adventure.  If the tech works, the Choose Your Own Adventure part is entirely new.  So the nice thing is cutting edge fresh.  The downside is, uh-oh.  So hopefully this goes really well.  So I'm going to talk about what Kubernetes is.  We're going to talk about how you, um, what you attack and how you attack it and then a little bit about how you defend.  We're going to do some demos.  Let's just start.  So the first thing I do anytime I talk about Kubernetes is talk about what the heck this thing is for.  And because, and I want to make sure I'm coming through on the mic.  I might come in through on the mic and like the back of the room.  Perfect.  Okay.  So what the heck this thing is for?  And, um, and you know, people are like, I know that this is hot.  I know that I should use it.  That's kind of the same way we all felt about Docker.  So, you know, why, why do we use this or what's made it so big?  And the, basically the, the Kubernetes came out of, in a lot of ways, it came out of what Google is doing with the Borg, right?  That's, it's an open source project that initially got released by Google and now has many, many, many contributors.  Um, but Kubernetes basically came out of this need.  We know that Google, we know that Google had just a, has a ton of computers and a ton of workloads to be clear.  You know, it's not exactly the most transparent company all the time, but they've had, they have some public statements they've made.  One is back in 2014, um, they were launching 2 billion containers per week.  If that were evenly distributed, it'd be 3300 containers per second.  Um, what's a container?  It's like a workload.  It's a program that's going to run for a little while and do something.  We don't know how long that while is.  But if you're launching 3300 a second, you can't have a human being decide which virtual machine or which host, you know, each program, each workload's going to go on.  So, um, you end up needing a tool for that.  Now, what else?  Um, well, they did this.  They were, they were launching all of those workloads onto 2.5 million servers.  At least that was their number back in 2016.  And in 2016, hard drive failure rates were just right below 2%.  So if every server had just one drive, and that's a big, you know, hand waving thing, but every, if every server had one drive, that meant they had 133 drive failures a day or one every nine minutes.  So now you've got, now you've got a resiliency problem, right?  Like you wanted to guarantee, you know, you wanted to guarantee so many nines of uptime.  And yet your physical machines are going to, you know, entire physical machines are going to die on some regular basis.  Like say, read notwithstanding, one every nine minutes.  So what features would you want if you wanted to be able to stage a ton of workloads over a ton of servers, figure out how to use the servers as the greatest, as close to their capacity as you could.  So you weren't buying lots of extra machines.  And you wanted to make sure that as programs crashed and as hard drives failed and so on, that you still had availability.  So you end up with what Kubernetes is, which is basically a software defined data center.  And we do that by container orchestration.  So what do you get?  So basically this is your feature set.  This is what you end up saying.  You end up saying, well, sometimes I have a lot of people checking their email on Gmail or searching right now for the thing that just happened in the news.  And sometimes I don't have as many.  So I'd like a whole lot of scaling.  I'd like that horizontal scaling.  I'd like to be able to say I have four copies of that workload.  No weight. I have 400. No weight. I have 50 and so on.  And you get automatic bin packing, which is that like which workloads should go on, which machines.  We can keep all the machines as full as we can.  And you get self healing like that hard drive crash thing.  And you get a whole bunch of other cool features.  You get like automated rollouts and rollbacks where you can basically deploy multiple versions at once and see if one of them gets, see if one of them ends up producing more sales or more of whatever it is you're trying to produce.  Or see if one's buggy and then be able to basically route all the traffic to the one that isn't and so on.  You get other kinds of things you would need if you're going to do microservices.  So you get service discovery and you get load balancing.  And then you get orchestration of your storage and you get secrets and configuration management.  So that's kind of, that's what the purpose was.  I'm going to do a little bit more theory.  So just kind of like what's a pod?  What's a container?  So right, we, when we're talking about Kubernetes,  we're talking about containers, but we say pods a lot.  And basically pods are the smallest unit of computing Kubernetes.  So all the containers in a pod are sharing an IP address.  The only reason you put multiple containers in a pod is if you want them to absolutely positively land on the same host.  Like they have, they have need for very, very quick network access or what have you.  Maybe they need the same local storage.  An odd thing about pods is all the containers in a pod share an IP address.  Okay.  So then you've got Kubernetes nodes.  Simply put a node is any system you're going to run Kubernetes on.  A node is any of the, any of the systems in your cluster,  whether those are virtual machines or physical hardware.  The nodes run a, well, first and foremost, the nodes to be a node, you've got to,  you have to have a container run time.  So you have to have something like Docker run, see, or what have you.  Something that's going to basically take instructions about,  take instructions from Kubernetes about creating containers and create them and maybe destroy them and so on.  You also need a kubelet and a kubelet, Kubernetes kubelet.  A kubelet is just this thing that runs on every Kubernetes node and it ties the,  it ties the container run time or the node into the rest of Kubernetes.  It talks to the API servers and so on.  Then you've got a kube proxy and that thing manipulates IP tables rules to create cool networking.  It also serves as an actual proxy.  So, yeah, we'll get into that a little bit more.  And then services.  Services are big because in Kubernetes and services give me a great occasion to kind of describe how you talk about the names of things in Kubernetes.  When we're doing the attack demos, we're going to be talking about this pod and this pod and this pod.  But for the most part in Kubernetes, you don't ever think about which pod names you want, right?  It's, the whole idea is that the individual pods, right, the containers, they're very, very much supposed to be cattle.  They're not supposed to have names.  They're supposed to be interchangeable.  I'm supposed to be able to trash them, start new ones up.  I have to, I'm going for, I'm going for that.  So what does that mean?  I mean, if I want to talk to a pod, what do I do?  You don't.  I mean, you might, but rarely, mostly you talk to a service and a service is mapped to multiple pods.  Think of it as an internal load balancer.  Now, now, if you've got, like, why am I, why do I say that would get me into something cool?  That gets me into labels.  And basically, you have a load balancer.  You can't say this load balancer should be pointed at these five pod names, right?  First, you're going to have a whole lot more.  They're going to get added.  Second, those pods are going to die at any moment.  Third, you're trying to go really, really fast.  Remember, you know, not 3,300 containers per second being launched, right?  So what do you do?  You end up basically labeling everything.  So you label all the pods.  You give them labels, and those labels are basically, sometimes they're by convention,  sometimes they're whatever you chose, but you label the pods and then you create a service.  And you say everything that, you know, this service is going to be a name, an IP address, a port,  and everything that goes to that name and port or IP address and port is going to be sent to one of the pods  that have the set of labels the service specifies.  That does kind of open an interesting question.  Like, if you are allowed to stage pods onto a cluster, could you start intercepting other people's traffic  by just giving yourself the same labels that their services are looking to land things on?  Okay, interesting.  Last thing we talk about, last big, big, you know, kind of simple glossary concept we need is namespaces.  The namespaces are just a logical grouping for things.  So we'll see those.  They're often used, they're often used to separate projects or to separate teams, to separate an individual,  you know, maybe to separate, well, anything that you, anything you choose.  So there's some way of chopping things up.  And it's important to say Kubernetes namespace is not the same as a Docker or Linux namespace,  so much so that when I teach classes on this, one of the other folks in Guardian, Jared Fredes,  asked if we could please call Kubernetes namespaces, space names, since Docker and Linux, you know,  already have namespaces.  So it was, you know, don't overload the term.  So here's a quick glossary slide.  I'm going to say a tiny bit more on theory and then we'll have some fun.  So, so Kubernetes is basically works in this declarative model.  You don't generally say, you're going to see me generally like, hey, do this thing right now.  But you don't generally tell Kubernetes to do a thing right now.  You don't say, hey, take this, you know, take this container image, start a container on that note over there with it.  Right.  You tend to say, you know, it's very, it's very delegating and hands off, right.  So the idea is you say, Kubernetes, I have this program.  I've created an image, maybe I've placed that image on Docker hub, maybe put it on some other container registry.  And I'd like you to now like download that, like have some nodes, download that image.  I'd like at least say five to be running it.  Like if you need more scale up, but you know, I want five, no less than three, like let's, you know, just do me a favor and get that running.  And Kubernetes says, sure, and it's got a bunch of like control loops that basically first make that the state of the world.  So you describe the state of the world you want it, it basically spends its whole life trying to keep the state of the world where you want it, where you've said it has to be.  And then it basically loops and it says, ah, a machine fell down.  I guess I'll have to reassign those workloads or looks like something's not responding.  I'll have to, you know, I'll have to, maybe I'll kill that off and I'll start something else.  So you describe the state of the world.  I think this is basically my last theory and I hope I haven't given you too much.  So the kind of target components, there are a bunch of components you'll target.  The big ones that you hear about are especially on the first two slides here.  So you'll target the API server, the thing that everybody's basically doing all of these HTTPS and God forbid HTTP requests against and asking to do things.  And the cubelet, which we said bridged the, bridged each node into the system.  The container runtime, you can often think of Docker and I'll often say Docker, but there are many, there are multiple container run times.  And that container run time is the thing that pulls the container images from the registry, you know, like Docker Hub or what have you, and instructs the kernel to start up a container.  You have an, you have an LCD server that maintains all the state.  And if you ever get access to that thing, yeah, there's not really a lot of chopped up authorization.  So if you get access to it, you kind of own all the, you own all of the persistent idea of what the system is.  And so that's very good for you.  A Kubernetes dashboard, that was the thing that got a, as I recall, that was the thing that got Tesla in trouble where their dashboard was available to the internet.  And I think was allowing anonymous access to do anything.  But that's a web interface as I, I think it's being deprecated.  There's metrics components and different kinds of cluts, cluster services that'll give you good information.  Those, those are your usual targets.  Just to throw in two more, two more components that people that it's really helpful just, you know, if you're operating Kubernetes, defending it or attacking it are useful to know.  One is the controller manager and the controller manager, it makes it sound like it's like the controller manager would have lots of separate controller programs.  And it kind of does, but they're all just compiled into one big, you know, just compiled into one big program.  So the controller manager has all of the controllers, the controllers of things that basically do those loops and say, is everything the way it was supposed to be?  Is everything the way it was supposed to be?  Ah, something's different.  I'm supposed to have five copies of this.  I've got four.  I'll go talk to the next one, the scheduler and say, excuse me, scheduler, can you find somewhere to, can you find somewhere to put this?  Can you find a node we can put things on?  We tend to target these components only through the API server.  So attacks on Kubernetes, this is where we start to finally get to attack and defense.  Attacks on Kubernetes, they start from, they start from the perspective of compromise pod.  So you can attack, you can attack a Kubernetes cluster from across the network and it's even less effective than attacking most companies, you know, across the internet.  Across the internet has been, right, lately.  We've gotten pretty good at parameters.  Kubernetes is pretty darn good at parameters because for the most part, there's no traffic being allowed in unless it's had some kind of ingress defined for it.  So for the most part, you attack a Kubernetes cluster from the perspective that you are in a workload, that you're in a container and that container is compromised or otherwise under your control.  So what are the ways the attacker might end up with that kind of control?  One is, yeah, as you'll see in our demos, this is the vector I like to show.  It's the attacker found a vulnerable thing that you were hosting in Kubernetes, got remote code execution and now they're in a container.  And from that container, they may have the ability to do much more far-ranging things because we're in an orchestration system, a very automatic one.  They may be able to do far more wide-ranging things than if they just busted that, you know, than if they just busted a, you know, some web server or something that was in a cloud environment or in a data or in an on-prem data center.  So how else would the attacker have gotten in?  The attacker might have just fished somebody.  Brad Giesman, who I'll, whose name I probably just butchered and whose Twitter handle I'll have at the end of these slides,  has a great slide in one of his presentations about how Kube control apply and that a URL is the new curl URL into shell, right?  If you, one of the great ways to fish somebody would be if you put up a how-to article on Stack Overflow or what have you, and the, you know, part of that article was, oh, just apply my, you know, apply my YAML from this URL.  You can get anyone to apply the YAML from the URL, then you've got places for your crypto mining to go.  So, and our attacker could just be some internal threat actor who's looking to escalate their privileges.  I often, you'll, you'll often hear me use the word user and attacker interchangeably on the defense side.  That's not really, because I think all my users are hostile.  I just don't know which ones of them might be.  I don't know who's being coerced.  I don't know who's, you know, getting curious.  I don't know who's actually got an axe to grind.  And I think we all know these privileges are a good idea at this point.  So, okay, so, so if an attacker is going to be in a pod, what are they going to do with that?  I'm going to give you a few, I'm going to give you a few things there's even more, I'm sure.  So, this is kind of the general way.  So they could use just the network access that they get provided by the pod to access other services.  And I'll say very, very, very often, especially in the microservice model, a Kubernetes cluster has a bunch of things that are all supposed to talk to each other.  And it's actually, it can be a little difficult or at least take some work to sit down and say, okay, what should and what shouldn't.  So once you're in the cluster, you have access to a whole bunch of things, to a whole bunch of things, many of which might not be intended to be communicated with by anyone on the internet or anyone in the, anyone at the service that you've pond is trying to serve.  So what else? The attacker could attack other containers that are in their same pod.  Multiple containers in a pod, they share, well, I told you, they share an IP address, they share a local host.  So it's a cool little Linux namespace thing, which we won't get into unless somebody asks me later on.  What else? So the attacker could make requests to the API server or to a kubelet and ask those to maybe run commands in another pod, maybe even interactively, to start a new pod that has privilege.  And that privilege could be all kinds. It could be a privileged pod, which means it'll start on the host and have access to system devices and be able to, among other things, maybe mount the hard drive, a little of the node.  But there's all kinds of privilege you can get out of starting new pods, and I'll show you that.  They might gather secrets that Kubernetes provides to pods because Kubernetes has a whole thing where it needs to provide configurations to pods,  because you're really supposed to make those pods come up with a static image.  So the idea is the pod starts with a static, totally replaceable image and pulls whatever information it needs from elsewhere.  You might connect to the Kubernetes dashboard to perform actions.  Maybe it wasn't accessible from outside the cluster. God, God hope. Or gosh, hope.  What else? The attacker might find a way to interact with the LCD server and change the cluster state.  It's a nice way if there are network access control rules, if there are network policies in place that are getting in your way.  If you can talk to the thing that stores them, yeah, you could potentially change them.  And the other one, the one that's really fun to me for anything that's either in a cloud provider or even in a non-prem cloud,  is that the attacker might interact with the cloud service provider, because many workloads will one way or another,  and I'll show this to you too, will have privilege on the cloud provider.  So I'm going to show you a bunch of demos, and these demos are real.  They're often things that we do in classes I teach.  They're also, I'm going to tell you later on, we have an open source project,  and it's an intentionally vulnerable Kubernetes cluster. Call it like damn vulnerable Kubernetes cluster.  This one's called Bustacube. We've got an Easter egg Bustamoo video that you can find if you go to the bustacube.com website.  But so some of the things I'm showing you are basically available as scenarios that you can play in this open source free damn vulnerable Kubernetes cluster.  So I've done a thing where I've made my videos part of the slide, and so now I'm going to see if that actually works.  Please work.  And if it doesn't, we'll have to use YouTube one way or another, or something else.  Okay.  Color me very embarrassed. Let's just see if this one's the one.  You should always test your demos, even when you record them.  So I tell you what, we're going to find another way to do it, and I am going to, let's see, I'm going to unplug from the projector for a second,  because that means I'm going to go and look at my own file system and I'd like to not share it.  And here we go.  Just embed the videos in the slides, they said. It'll be fun, they said.  So I'm pretty old enough to have played Warcraft and you click on the guy and he's like, what? And you click on him again and he says something else and he says, you know,  join the army, he said. See the world, he said.  Okay. So here's our demo.  And hopefully entering, oh, well, that's not a demo.  Is that a demo? That's not a demo either.  Anybody seen my screen? There we go. Let's just hit this on mirror and call it a day.  Okay.  So that's not the right.  Don't mind me, I'm clearly a noob.  I may need alcohol.  So this is one of the Bust-a-Cube scenarios.  So you go up and you see a WordPress site and the WordPress site, I will tell you, is hosted in Kubernetes.  And you fire up maybe Durbuster to go looking for hidden content and Durbuster takes a little bit of time, but we can run it and let's go a little faster.  And what we find is backdoor.php.  So when we surf to backdoor.php, this is what happens.  I get this thing where basically this WordPress instance, like so many other WordPress instances, has been pre-compromised for your convenience.  So I type ID and it says you're dub, dub, dub data.  And hopefully I shut down Durbuster because it always choose a ton of resources.  It's something we always say in class.  It's running slow. Have you turned off Durbuster? No, it's 500 requests a second.  Well, that'll do something.  So I'm going to stand up a quick little web server in Cali because I found that reverse shell.  So I'm going to stand up a little web server.  And my purpose of the web server is to be able to hand a, to be able to hand a interpreter or a, you know, or if I wanted a, you know, whatever, some other kind of reverse shell.  But I'm going to go for a interpreter.  And so I just put in here like, hey, could you please pull this interpreter binary I've created from my web server and while you're at it, make it executable.  And while you're at it, run it.  And I like to fast forward my videos because who doesn't.  And go look at the, you go look at the web server and Mrs. Bin has been pulled the interpreter binary but also a copy of Kube control.  The command line utility that you use to do damage, I mean, to administer or use a Kubernetes cluster.  So I'm going to pull Kube control.  And the other thing I'm going to do is just take and make sure that it's landed here.  So here's Mrs. Bin, my interpreter binary.  Here's Kube control.  Cool.  And now if, if recorded J will type a little faster, we'll run the interpreter binary.  And I'm going to go over to my Metasploit console and actually be like, oh wait, I have an incoming interpreter that needs to connect to the console.  I should probably start the console.  This ordering is just great, huh?  But that's often how you work.  So I'll interact with that session.  And I'm now in a pod.  I know it doesn't look like it.  It looks like an ordinary Metasploit, you know, Poned Metasploit instance.  So, but I see my copy of Kube control.  I'm happy.  And when I run mount, I get a tremendous number of file systems.  So I'm not looking at one hard disk or what have you and a bunch of them put there by Docker and by Kubernetes.  And so the one that's really interesting to me is this one that says run secrets Kubernetes service account, IO service account.  As soon as I see that directory mounted, I know that I'm going to have some level of access to the API server and I'm happy.  So I'm going to go and first I'm just going to go and collect my flag.  I built this scenario as a capture the flag.  So let's look at our little flag.  And it says, hey, you've found the servers back door.  You might notice this isn't your usual Linux system.  You're in a cluster, my friend.  It's time to bust a Kube.  This container doesn't have any more flags.  Can you find your way to another pod?  And tint.  So cool thing in Kubernetes is, like I said, one of the things you can do is see if you can move to another workload.  So you're in a container.  Can you move to another one?  So I'm going to see what my service account can do.  I'm going to go and look in that directory and that Run Secrets Kubernetes IO Service Count Directory.  If recorded J will type just a little bit faster.  And I look at that namespace and it says you're in the marketing namespace.  So that directory has a token.  It has the certificate authorities public cert.  And it has a description of what namespace you're in.  And that namespace, like I said, is kind of a wave organizing.  So we're in the marketing namespace.  We'll come back to that later.  And then I'm also going to go and look at the environment variables that get pushed into every pod.  And my environment variables will tell me where to find the Kubernetes API server.  So there's this assumption in Kubernetes that every workload will need to ask Kubernetes to do things.  I don't know if that's really the greatest assumption, but I think let's just call it not the strongest of defaults.  So I've got the API servers IP address and port.  And I'll clear my screen and I'm going to set up a quick little kube control where I'm saying alias kube control to the path to kube control.  The token pasted in from this long path to the token, the certificate authority path, the namespace and the server that I just got.  And so every time I type kube control, all that stuff will be passed in.  So let's just kind of skip and let's see what did I do.  So I said, let's do a quick get pods.  And get pods is honestly my go to just to figure out if my setup is working.  And so it says, oh, there's two pods.  You're the WordPress one.  And I ran host name and it says, that's the one you're in.  And then the WordPress my SQL one.  And that's another one.  So let me see if I can move over to it because that was what the hint said.  So I'll do kube control exec it looks a whole lot like Docker exec it.  And what program I'd like to add to that container.  I'm going to add a shell.  I get this kind of error, but I'm in the container.  I've just moved over and I've probably moved over to a different host.  Like I'm probably somewhere else in the data center now.  So there's another flag and we'll go get our flag.  And I can't clear my screen because my terminal is not set.  So I set it to VT 100 because that was one of my first back in the BBS days because I'm exceedingly old.  And it says, hey, you found your way to another container.  Actually, you're on a whole new virtual machine.  We're moving from one container to another in a Kubernetes system, often mean switching nodes.  Your end goal is to get to developers non-containerized host.  So it's to get not just to own the cluster or potentially, but to get to a machine that's not in the cluster.  So to get there, you're going to want to see if you can hop tenants.  Tenants are usually separated by namespaces.  So let's see what we can do.  I know I'm fast-forwarding through my stuff, but so I'm in the other, I mean, that MySQL want to try to get pods.  And the problem I'm running into is like, hey, you're not allowed to get pods.  So the cool thing is I'm in a different service account.  I was in the marketing WordPress service account.  It's allowed to get pods.  The marketing MySQL one can't.  So over time, I might want to like go around and pick up all these service accounts and like hold on to them.  Remember, it was just tokens.  So if I can store all the tokens, I'll have like, I can end up building privilege wider than just what one thing had.  So I did get nodes and I found out the IP address for node.  Here's a name, but this node has an IP address.  This is, this one is, this one's running in the cloud provider.  And there's a cool thing.  So you can talk to the KubeLit and this is, thank gosh, finally not a default, but some clusters will have it.  You can talk to the KubeLit on, it's got a couple ports, 10, 250.  I'm going to go and say, can you give me a list of running pods?  And it gives me JSON because everything in Kubernetes is JSON and YAML.  So I'll take that and I'll run that again.  And I apologize for this big block here.  So I'll run that again and I'll add some JQ, JSON query, really good tool to learn.  And I'm going to say, give me all the pods, kind of better format it, give me their name and their namespace.  And what I'm looking for is to see if there's a pod that's in, that's in another namespace besides the marketing one.  And a quickly fine one.  There's a dev pod.  And that dev pod, so Kubernetes loves to tell you about anything you want and that's really, really nice as an attacker,  especially if you can script it.  So I've got dev sync, I've got dev web.  These are two containers that are running in the same pod.  And I've got the image they came from, so they're up on Docker Hub, you could pull them yourself if you wanted.  And as an attacker, that's often what you're doing is basically saying, well, what workloads are running,  and you can go look in container images and see if there are any secrets or anything that would be really helpful to me.  It's very, very nice of the defenders to hand you descriptions of their workloads.  So I'm going to look at that, I'm going to run another command, and this is, I did running pods.  I'm going to tell the cooblet, hey, would you mind just executing a command for me?  So I'm going to pass it run, and then the namespace, the pod, and the container name.  And I'll do command equals ID.  And I get an error code that says basically no, either there's no shell in there or I'm not allowed.  I will tell you that there's no shell in there.  So, but remember, there were two containers in that pod.  So I saw dev sync and dev web, and that's a really common thing you see outside of containerized environments,  like you have a web server, and you have something that's there to pull it content in  from wherever that content's being created or stored.  So I was hoping I could get to dev sync because it'll have that, it'll have the credentials for pulling content.  It'll have an SSH key. I couldn't, so let me try dev web.  So dev web, I can run commands in.  I ran ID, it says, yeah, that thing's even running as root. Very nice.  And so let's try a different command. Let's look at the file system.  Oh, look, another flag. I like flags.  I love doing CTFs, and I love doing CTFs actually like first doing them as an attacker  and then going and basically doing them as a defender.  It's saying how many, how much of the attack can I break with proactive stuff,  with stuff that you could have thought of before you knew that you had the vulnerability.  So this thing says, hey, you're not in the marketing department anymore,  you're in the development department's namespace, so can you go take over the development department's non-containerized machines?  And they're big fans of the matrix, so it's named the mainframe, named for the machine's mainframe  that may or may not have been in the second and third movie depending on your religious beliefs.  There are many people I know who acknowledge the truth as they believe it,  that there were no second and third matrix movies.  Only quests for more money.  So here we go. So we've got, we're trying to do a get pods.  So I am, I'm in one pod asking the cooblet on another host if it mines running commands in another pod,  and I'm asking that pod to run coop control commands.  I find out, okay, dev app isn't allowed to list pods, but it's kind of cool.  I now know that I've got a third service account. I've got, I've got the dev dev app one.  So I'll try get secrets. Let's see what secrets are available.  Oh, nice. There's a machine, there's a mainframe login and an SSH key.  Well, the two pods in there, the two containers in there are going to likely have access to the same secret.  They've been placed in the same place. They don't necessarily need it in dev web, but what the hell?  Let's go and ask for a copy of that secret.  And I promised the next demo is hackers themed and choose your own adventure.  So we're going to have some fun.  So here's our SSH key and you're like, that doesn't look like an SSH key, and that's because it's a secret.  It's, it's opaigly encoded. It's base 64 encoded.  So we can decode it and boom, there's our SSH private key.  And we can do the same thing with the other secret here.  So we can find out basically what is it used for.  So I'm just going to fast forward a little bit and tell you that it's the, that it's a user, it's a user and a host in AWS,  but not part of the cluster.  So now we could SSH into that and we're in the developer's mainframe.  And it says, hey, you made it to the machine mainframe.  Can you, for extra credit, can you get credentials to take over the whole cloud environment?  And, and we can do that, but I'm going to set the rest of that aside because we do that in another demo.  So, so let's see.  Let me go back to the slides for a second.  And here was our, and I'm sure that I'll be, because the vagaries of having the videos, you know, in a separate program,  I'm sure I'm going to be doing a little bit of going back and forth and having the whole screen do this.  So here's our attack path.  We got a interpreter in the WordPress container.  We moved into MySQL container with the kube control exec.  We used the MySQL containers totally unfettered network access inside the cluster to reach a kubelet that we wouldn't have been able to reach from outside the cluster on the master node.  And that kubelet let us basically take a look at a, take a look at, take a look at run commands in another container in another namespace.  And from that other container in that other namespace, we found, we figured out there was a, there were secrets.  There was an SSH key.  So we got that SSH key.  We authenticated to the high value developer machine and later on we ended up getting to take control of the AWS account.  And we're going to skip that right now because we're going to do that in a much more fun way.  So we're going to play hackers choose your own adventure.  And I'm going to drink some water for a second because whoa.  So I like the movie Hackers.  It's a, it's a fun movie.  It is an oldie but goodie.  It is thoroughly realistic.  And we all know that whenever we're hacking, we, we see like strange translucent 3D models and, and you know, of buildings and we fly through a city and it's really cool.  And all of that was on, was clearly done with the technology that had Apple twos and floppy disks.  So, I mean, I remember 3D rendering back then.  I wasn't doing it.  I was watching people do it.  And yeah, no, not on that.  So here we go.  Let's start here. And, and again, this is where I have embedded video.  So we will, unless I click on that and it works.  Let's see.  Oh, hey, look at that.  Now can we full screen that?  Is there a way?  No.  So we're going back to oldie but goodie.  And here we are.  First, this is going to look a little familiar.  So we've got a, we've got, what's that?  Oh, you're on the slide and I'm on the, and I'm on the video.  Okay. Boom.  And switch.  Come on.  I'm going to Windows computers soon.  And that's a lot for me to say, trust me.  So anyway, here we go.  So we've got the, we've got the Gibson control room.  We have Penn from Penn and Tower, who's apparently a techno weenie according to Ben from Short Circuit,  who somehow was playing in Indian and talking about Mr. Johnny Five.  But Mr. Johnny Five here was the plague.  And we have this nice little form that's called the Gibson control room.  And the Gibson control room clearly lets you like leave a message.  So it's based on a, it's based on a actual, on a Kubernetes documentation vulnerable app.  They didn't mean for it to be vulnerable.  But it's only a little vulnerable.  So I've added a vulnerability to that.  So the, so what we've got is I look at the source on that page and I find that the form is basically being provided by some jobs.  So if I go look at that JavaScript, what happens when you submit the form is you're, you're referencing guestbook.php.  You're sending to guestbook.php.  You're saying, hey, command set key messages value this.  Okay, really?  You're going to let me just, you're going to let me decide what, what key I'm going to set.  You're not just going to have me set whatever key, you know, was meant to be set by this form.  So that was their bug.  And what I've added to it is I have something that actually runs code out of a different redis key in this, you know, back in redis database.  So we go and take the Gibson control room.  And if we were to type a message, you know, it shows up, this is the intended, this is the intended purpose.  But instead, we're going to paste that same thing that happens when you submit the form, except instead of the message key, we're going to try a different key.  Or we'll try a, let's see how it works.  We're going to hit foo.  And we'll put in just some key, some random key.  Oh, cool.  Okay, so JavaScript is probably updating that key.  Let's try going to status PHP.  So I'm asking you to imagine that we did a derbuster or looked at robots.txt.  We found a status PHP.  I'm not doing it because, well, I'd like to spend our time on Kubernetes.  So I go to the status PHP page and says, hey, no command to run.  I'm going to set the reddest key, the reddest command key to this so that you can reload the page and that will get run.  So if I reload the page, I get the output of, you know, I get the output of scrolling that web server.  It says all services are operational.  Here at, what was it, Ellington mineral?  Ellington mineral, Ellington, I think.  So as long as there's a command key, let's go and set it.  And I've set my command key now to environments, to run environments, like we did before to find a Kubernetes cluster to find the Kubernetes API server.  And here we go.  So we know that once we're in a workload, if we reach out to 10.96.01, we'll get to the Kubernetes API server.  So I'm going to go a little further, kind of similar to last time.  I'm going to curl in.  I'm going to tell this thing, excuse me, would you just pull in a interpreter binary I created and connect back to me?  So I have it pull it in.  I set that key.  And now when I run status, the page hangs.  I love when pages hang when I've just given them binaries because it generally means the binary is running successfully.  So if I go over to my terminal where I've got a menisploit handler waiting this time, I find there's a session waiting for me.  And the first thing I'm going to do before I start interacting with a shell there is I'm going to upload, I'm going to upload a copy of kubectl,  which I've stored on this Cali system in this directory.  So I'll upload kubectl and I'm going to upload some YAML, which we'll get to later.  So I've uploaded some YAML, which we may or may not get to later depending on your choices.  And I've asked for a shell, had PWD.  Looks like my video is cut off by one character.  A hit ID, it says your www.data user.  I try SSHing.  Well, I don't know what I tried because that was the end of that part.  And now we're going to do this part so that we're a little bit gentler.  I'm going to ask you, so do you want to try the YAML files that I put up were ones that would mount a volume on the host?  Like they try to mount the host's own directory, the old docker, start a container mounting the host.  So do you want us to start the host mounting pod and see if we can take over a node?  Or do you want us to exec into another pod?  I heard the host mount first, so we'll try that.  So let's see.  Boom.  And boom.  Here we are.  We're going to run that front end.  And we'll set kubectl executable so we can run it.  We'll go and set up our kubectl.  We'll try to run kubectl apply.  Realize, wait, we've got to set all those parameters we set before.  Sometimes you get lucky.  And so I'm going to take that same directory, that same run secrets kubernetes.io service count that's being mounted into the pod.  I'll set a variable called dir to that, export dir to that, just to make this a little bit easier to type, a little more manageable.  So if I look at the dir, got the ca cert, namespace token just like before.  Let's go and run.  alias kubectl is going to mean that, same way we did before.  So servers out on 10.96.01.  That was what we found from that environment variable.  And now let's look at our alias.  That's that token being pasted in.  So kubectl get pods.  Cool.  We've got a front end.  We've got front end pods.  We've got a redis master.  We've got an Apache status, that thing we saw before.  We're in the front end.  So let's try launching a pod.  kubectl attack, so kubectl apply.  I like it as kubectl attack pod.  And it says, sorry, your service count front end isn't allowed to create resources.  So, and this is where I get to my slide.  So we'll do that real quick.  I was considering for this one, oh, damn it.  Let's just keep the screen mirrored.  So I was considering for this one, doing a spandex quote from the movie,  and I decided that wasn't a sentiment I wanted to voice.  So I see that I'm going to make sure that I'm moving on time.  So at this point, we'll go back to our original,  we'll go back to the other thing we could do.  So our two options, where do we want to launch a pod or exact in the back end?  So let's exact in the back end.  And, or let's try exacting into another pod.  So we'll go like we did in our other demo, we'll try exacting into a pod.  Keep wishing we had a better terminal.  kubectl get pods, I see that there's a redis, there's a redis, that redis setup.  Remember, I was pushing redis keys.  I might try to investigate the redis and see what else now that I've got more direct access to it,  because redis is very often, well, almost always unauthenticated.  But let's go into the redis pod.  So I'm rooting there, I'm redis master, I'll grab a copy of the interpreter binary I've been using,  just because that's a very convenient way of putting files into something.  And I'll background that, and I'm going to switch over now that I've got the redis master,  that pod running my interpreter.  So I'm going to switch over and catch that session.  And so in that session, I'll go and upload the same kubectl that I did before.  I'll also upload that attack pod that I just used before.  And I'll also add something called attack demon set, just in case we need it.  So, now we get another choice.  So you're in the back end. Do you want to try host-mounting pods,  or do you want to check out the cloud APIs?  Cloud APIs? Cool.  Okay, so here we are.  And I'm going to show you this in two ways.  And can I get a rough time check?  How much time will you let me be late?  Oh, less than five.  Okay, so this is where I normally kick on the speed control.  So, okay, so we're in a pod. We're in that redis master pod.  And we say, hey, I'd like to talk to the metadata API.  So every cloud provider gives you 169254, 169254.  That part's common.  What the calls are that you make against it, they document wonderfully,  and you read the documentation.  So this is on GCP.  So I'm going to go and say, can you tell me what the project ID is?  I need that project ID to do API calls.  But then the more interesting one is usually where you say,  would you mind, in addition to the project ID,  which I'm going to set into a variable,  would you mind telling me what service accounts are assigned to the machine I'm on,  so the machine this pod's running on?  And there are two, but default is mapped to that other one.  So I'm just going to use default because it's easier typing.  And I'm going to go and say, okay, for that service account,  would you do me a favor and just give me a bearer token that I can use to act as that service account?  And it says, sure.  Here's your token.  It expires in like a half an hour or an hour, but I've got a token.  So I'll set a variable to that.  And now what I'm going to do is say, let's go and I've set that token to that.  And I need to refresh that token so most of my commands will pull that token over and over, every command.  So I'm going to use that token, type out authorization bearer token where the token's provided,  say I'll take JSON, and now I'll say, hey, for this project, I'd like a list of buckets.  And it says, oh, there's only one.  Okay, well, what's in that bucket?  And so I basically say, give me a list of objects.  And the way I do that is it's like slash B for a list of buckets,  then it's slash B, that bucket, slash O for a list of objects.  And then for those objects, I can run through them.  Now, on this one, I don't find anything that's all that useful.  I don't find what I find in some cloud environments.  I'm going to show you in another demo somehow.  But I do find a kubion v.  So I can pull in a metadata instance for this cluster that was set by the installer.  And when I do that, what I get out of it is, among other things,  a couple service count tokens for the cluster.  And that ends up leading, which I won't now show you, but that ends up leading to win.  And so just for the end of our, for the end of that, that then leads to our crash and burn.  So if Ming will let me have one more minute, I will show you a cooler version of that.  We have a tool that we keep soft releasing, but never kind of like putting up on Twitter.  So it's called Parattis.  And Parattis will, if I actually just, I'm just nervous.  So it's in here, but somehow I'm, let's do this.  So Parattis is basically a remote access Trojan for Kubernetes.  You run this and it does all kinds of things.  And I'm going to start this from the beginning, but it is under two minutes.  And then I will get the heck off the stage and tell you that you should ask me for stickers.  So it's got a bunch of things.  One of those I had said, okay, 13, go get the IAM credentials from GCP.  The same thing we just did, get the cloud service account.  And then, come on, Jay, type faster, highlight faster.  So that's for that same one I just got.  And now let's take that and let's pull, let's go looking in GCS in the storage buckets.  For service count tokens.  Oh, look, this installer, this one was cops, in left service count tokens.  There for all the components, including a cluster admin, which would be perfectly useful.  And so that's the like, basically, that's probably roughly what happened to Capital One recently,  where it was talk to the metadata API, get your cloud account,  use that to go rating for information or access.  So anyway, so now once I've done that, Parities will basically store every, every count it gets.  And then you basically say, okay, I'm going to switch context.  And you go and you go through and you collect all of the service count tokens,  and then you basically have this big old store that's your loot, and it keeps it in memory.  And you run around and basically get more access and get more access.  We've even got a, this video will be up on YouTube, but we've even got something that basically takes over the nodes.  If it's allowed to mount a host path volume, it'll go and add a cron job that will send a netcat shell back to your,  back to the IP address of your choice.  So once we have this cluster admin, we're allowed to, we can go and do everything which included like,  hey, now show me all the secrets.  And all the secrets includes all of the tokens, all the service counts for every component.  And, and Parities will not just grab, will not just list these and get them,  but it'll also store them and let you use them.  So it's a fun little pen test tool, and it's up on GitHub.  We'd love some help, we'd love some help producing it.  So it's an inguardians tool.  We do a ton of Kubernetes work in inguardians, and I will now get off the stage and take questions all the way out in the hallway.  So if you see me, please ask me for a sticker.  Thank you.
Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_0_1583.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_1584_1715.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_1716_2567.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_2568_3431.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_3432_7475.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_7476_9467.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_9468_15023.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_15024_32771.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_32772_33707.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_33708_34475.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_34476_34535.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_34536_34691.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_34692_35015.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_35016_35255.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_35256_35411.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_35412_35927.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_35928_35975.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_35976_36119.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_36120_36311.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_36312_36395.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_36396_37031.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_37032_37595.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_37596_38255.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_38256_38375.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_38376_39611.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_39612_39671.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_39672_42191.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_42192_42995.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_42996_48611.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_48612_54887.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_54888_55415.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_55416_55463.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_55464_55667.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_55668_55823.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_55824_58199.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_58200_58499.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_58500_59243.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_59244_59411.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_59412_59459.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_59460_59579.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_59580_59639.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_59640_59759.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_59760_60263.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_60264_60515.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_60516_61859.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_61860_62471.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_62472_62531.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_62532_62891.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_62892_63491.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_63492_63575.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_63576_64451.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_64452_65603.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_65604_65807.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_65808_66803.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_66804_66923.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_66924_69359.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_69360_69575.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_69576_69707.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_69708_69767.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_69768_69923.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_69924_70007.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_70008_70367.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_70368_70415.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_70416_70487.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_70488_72515.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_72516_72875.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_72876_72947.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_72948_73007.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_73008_73055.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_73056_77123.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_77124_77243.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_77244_77687.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_77688_77735.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_77736_78407.png Jay_Beale_-_Attacking_and_Defending_Kubernetes_-_D_2fmAuR3rnBo_78408_82250.png