Fundamentals of Kubernetes for Django developers
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Dumpleton, Graham
A hot topic in recent years is running applications in containers. Tools such as docker and rkt have made deployment of applications using Linux containers easier, but they do not alone provide everything that is needed to easily manage many applications, or run containers at scale across a cluster of machines. In this talk you will learn about the fundamental concepts and terminology of Kubernetes and why it is emerging as the de-facto standard for container orchestration and scheduling. The talk will step through how to deploy to Kubernetes a Python web application, implemented using Django, as a way of seeing what happens under the covers when you do so.
 If you're not familiar with who I am, I am not a Django developer. I know about the internals of Django, I haven't had a deploy it, but I don't actually have really had a right Django apps. So my history is I am the developer of modWizki for Apache, for hosting applications with Apache, also a response for a Python package called wrapped, which is for implementing decorators, and you may also know of me from Nurellic, where I wrote the Python agent for Nurellic and do lots of evil things to your Python code. I don't work at Nurellic now, I currently work at OpenShift, Red Hat, where I am a developer advocate for Kubernetes and OpenShift. And I always have problems with my pointer. We'll go to backup plan. That's better. So Kubernetes, I don't know how many here are familiar with Kubernetes. Anyone want to put their hands up and know about it? So for those who have got into Kubernetes already, would this be a fair explanation, would you say? Yes, more tears. So Kubernetes, it's an up and coming thing, but it's a very large ecosystem and can be very, very, very confusing when you start to get in. It's a very large learning curve. If we look at the landscape of what exists around Kubernetes, there is a lot and a lot of stuff now starting to come around it. They are software packages you can use with Kubernetes, but also all the different companies are bringing stuff to it. So there's a lot in there which potentially could learn. So it can, depending on how deep you go into that, you can get very, very complicated and it's very, very mind-blowing. So if you are getting into it, be patient because it can take a long time to get up to speed on it. To make it easier rather than just dive in and think, okay, I'll go and get original Kubernetes distribution and start using that, I strongly encourage you to look around at easier ways of getting into Kubernetes. And there's different ways there. We've hosted versions of Kubernetes and also distributions of Kubernetes. The way to think about Kubernetes, it's like a kernel. So whereas you've got a Linux kernel, which is the guts in there, which helps you to run up a patient on a Linux operating system, you're never going to install that kernel yourself. It's always better to go get a distribution. So you're always going to get a Ubuntu or Fedora or CentOS, Red Hat or other Linux distribution. So it's always better to go through that because they will provide you an experience which is going to provide extra bits on top of Kubernetes that's going to make your job easier to understand and use it. So think of Kubernetes as a bit like a kernel. And in this case, it's going to be the kernel for a cluster of machines where you're going to want to deploy an application. I'll say more about that later. So stepping back from that bit of warning about getting into Kubernetes and where you should perhaps go to start looking at it, what exactly are we talking about? We are talking about containers. We all heard of the containers now. It's the big buzzword from the last few years. This has actually been around for quite a long time. Google in their infrastructure has been using containers since about 2002, 2003. I can't remember exactly when. But even before that, it was the similar sorts of ideas of what you were doing, talking about containers existed in the form of Chirrut Jails in BSD and also Solaris Zones. So they're not a new concept. But containers are a thing that has come more popular of late, and it became popular because of a company which, when they announced it, was called DotCloud, but you'll know them as Docker. So this company called DotCloud at PyCon at Santa Clara in a lightning talk showed a really simple way of running up an application inside of what we call a container. So what is a container? This way to think about it is to compare in terms of what we've done in the past. If you're familiar with virtualization and the idea of running an operating system inside of Sandbox on your operating system, so I might have a Mac and I want to run Windows or Linux, I can install a version of an operating system, Windows or Linux, inside a virtual environment or virtual machine on my box, in a little Sandbox. And I'm quite happy to work away in that. And I can actually install multiple operating systems, multiple systems of Windows or a mix of Windows and Linux. And then inside of that, I can run an application. And it's providing a Sandbox, an environment where you can run stuff, and it's kept away and it's isolated from your host operating system, but also your other operating system running in a separate virtual environment, virtual machine. So containers, the idea behind them is a very lightweight way of running up an application inside of a container or Sandbox on a host operating system. Now the difference from traditional virtual machines is that in a traditional virtual machine, you are running a whole operating system inside that Sandbox. And then you're running your apps. So you've got a whole lot of application services in your system services and therefore that operating system, as well as your application. In a container, you are making use of the fact you've got a host operating system running under or underneath already, and you're going to run just the processes for your application in that Sandbox. So you're not going to run additional processes in there for system services. It's just the processes for your application. So with Django conference, so in this case, we're talking about Django web application. So you might be running your Django application under modWizGi, which I hope you are, but I know you're not. You're probably using UWizGi or Goonicorn, I'm sure. So all you're doing is you're running that WizGi server inside of that container with your Django application in it. But because it's in this containerized environment, it's still protected. If I get into that container, all I can see is those processes for that application. I cannot see operating system services in the underlying host operating system, and I can't see applications inside of other containers. So it's protecting things. So that's what it is. And it's meant to be this more lightweight thing, because we're not starting up a whole operating system. It's much quicker. You only need to start up your just your processes. Now to run off something in a container, what Docker did in terms of what they announced was they provided a couple of things. And it's all focused around tooling. A container, as I said, and not a new thing. They've existed for a long time. And the way you use them normally is you're using what's called Linux containers, or LACC. So there is this way of running up processes in constrained environments on the operating system. But that was always a difficult thing. Docker made it quite simple because they provided this program called Docker. And there was two important things for that. There was one, which was Docker build command. And the idea there is you could provide what was called a Docker file with a set of instructions of how to build an image which contained all of the bits and pieces you need. So you start out with a base image for the operating system. So it might be a 1, 2. And then you'll add your instructions on top of how to actually install your application with all the dependencies it needs, all the different Python packages, into that. And they all get built into an image. You want to then run that image, and we can run that up on our box. But what usually is going to happen is we're going to push that image up into what's called an image registry. Once you've got an image registry, we can then do Docker run, and we run it up. And away we go. So that's what they made a very simple process to do. Before that, using LACC was a lot of different steps to do all of that. The Docker run command, when we take that image, is going to do a lot of work to create that sandbox environment for you. But ultimately, even though we talk about Docker, it's still underneath using Linux containers. It's using the features of the operating system. Docker was just bringing the tooling to make that easy to do. Now running single containers with your application, we can do that. You can get it working. A bit of work to do, you can be happy. The problems occur when we start to want to run many, many containers, especially if we want to run across multiple hosts. Because we can only scale up applications, the number of instances or replicas of that application we were running on the single host so far. We're going to run out of resources. So we need to start to put them across hosts. Now Docker was great for running up an application in container on one machine. When we start to go beyond that one machine, that's when it starts to get more difficult. You have to start tracking. I need to run 10 instances. I can only fit five on this host. I have to run five here. I have to remember I've got five here. I have to run five over here. But then I've got all these different instances of them and they all are going to listen. They all expect to listen on port 80 for HTTP traffic. They can't all listen to port 80 because they're on the same host. So we need to worry about port assignments and put them all on different ports. But then you have to worry about, well, how do I distribute traffic? You have to start putting load balances across and distribute traffic. So very, very quickly it starts to become very complicated to be able to do that all yourself and track it. And this is where Kubernetes comes in. Kubernetes, as my kernel, as I mentioned before, is a means of managing the deployment of applications inside containers distributed across a cluster of machines. So it's a typical Kubernetes cluster. What you'll have is some master nodes. I've got one here. But if we talk about fault tolerant or high availability systems, you'll actually have three at least, an odd number. And you may have other infrastructure nodes in there which have other bits and pieces like the router. But I've simplified this. I've got one master node and I'm saying everything's running on there. That's the smart, main smarts of Kubernetes. That's where the database is there, which is tracking how, what we're deploying and so on. And also router and other bits and pieces. And then we all have our nodes. That's where your applications are going to run. So that's what Kubernetes is going to do, is provide that ability. Now, in this talk, I'm not going to delve down into the depths of how Kubernetes works. I'm just going to step through running a, deploying an application inside of Kubernetes using Kubernetes command line tool. And I'm going to use that as a means to just introduce different concepts of Kubernetes so that if you are going into Kubernetes, at least then you've heard what these concepts are once before at least, and they might make a bit more sense to you. So deploying. We've already built our image with Docker build, for example. We have it already up in an image registry somewhere. And a popular place that people do store images up, a place called Docker Hub. That was a public image registry which was run originally created by the Docker company, Inc. But there are other image registries you can run your own as well. So if I want to run up an image inside of Kubernetes, so if you're using Docker, I'd use Docker run. Here I'm just going to use kubectl run. It's very, very similar. I want to name that. So I'm going to name my application blog. I tell it where my image is and I tell it the port that I need to expose. If you've used Docker before, you'll see it's very, very similar to Docker run. The difference is that if I use Docker run, it's going to run it on my own machine, where the Docker service is, or the one you've logged into. Here, because we've got a cluster of machines, and I've just said, okay, I want to run run instance, Kubernetes is going to go, okay, I've got a cluster of machines I can run this on. It will go and work out for us where to run that and make a decision for us and get it up and running. When we do that, when we run Docker run, it's actually taking an action of doing that. It's actually going to essentially go right through and run it up and start it. Kubernetes works a bit different to how Docker is. When I run that Docker run command, it's not actually really starting things up. What it's actually doing is going to set up some configuration. If we actually run that same command and put dry run minus output YAML file in there, you can see what it's doing. What all it's actually going to do is create a YAML description of what it is I want deployed. That's actually going to be stuck inside of what's called a data store inside of Kubernetes, which actually uses an XED. That's actually all it's going to do. That's all the command does. Updates that database of information. Where Kubernetes works is what's called a declarative model. Rather than actually you do something that causes an action immediately, all you're doing is updating your configuration of what I want my application deployment to look like. Kubernetes comes along and is always checking these configuration objects, which exists inside of the database and goes, oh, I've got a new configuration object, or a configuration object has changed. I need to now go make the cluster or what you want for your deployment, what you have in the cluster, agree with what that configuration says. Because it's a new one, Kubernetes goes, good, oh, new deployment configuration, I'll go and do the steps needed to actually satisfy what your configuration says. In this case, our deployment, it was given a name. It was given some labels automatically to help us identify our configuration later on, which is very important. It had a number of instances, which was just one initially. We have a description of container, of what, and our image, which we're going to run inside of that container. So Kubernetes will go off and run up an instance of that for me. I've told it, I only want one, so it'll go and do one. I can then query back once I've done that. I can do a query back of the resources which we've created. You'll actually see my deployment there, you'll actually see some other resources created there as well. This is where my command only created one, the deployment. But Kubernetes has stepped in and said, OK, to satisfy your requirement of having your deployment, well, I actually need to go and create some other resource objects in there, more configuration. Now, the first one is what's called a replica set. It's the thing which is going to manage your instances of replication and so. So it pulls out of that deployment config information about the number of replicas you needed, the description of the container and the image that you needed to run. It's created a replica set for me. There's an aspect of Kubernetes which sits in there and looks for now for replica sets. When it sees that's created, it goes, oh, I need to now do something based on this replica set. So it pulls out the description of the container, it knows only one one instance, and it creates what's called a pod. Now, pod, I mentioned containers. Well, what's the relationship? Normally, when I say container, pod is the same thing. But in effect, a pod can be one or more containers. So when we talked about our description back there, you can see our containers in our demo description is actually a list. But I'm only describing one. So technically, I can have more than one container in what's called a pod. This name pod, where does it come from? Pod of Wales. And Docker, well, they had their logo of a whale. So that's where the naming come from. But you can more or less think about a pod as being a container most of the time. So now we've introduced a deployment. That triggers the creation of a replica set. And I've got a pod, which is my running instance of my container. If I need to look at what my application is doing, I can use that pod as the name of that pod as a means of getting out, say, logs and find out what happened when my application started up. But more importantly is if I want to scale it, what happens then? If I want to scale it, there's a command called kubectl scale. And I can actually refer to my deployment config for that. And I say, I now you want to have two replicas of my application. And all again, this has gone back. All this has gone back has changed that deployment config. Kubernetes has noticed a change in the configuration. It says, I need to now make the cluster match that. So it will go and update the replica set. Say, I've got two replicas. And then the controller for that, looking at replicas, kicks in and goes, oh, two, I'll create another pod for you. And Kubernetes is once again making decisions for you about where it's going to run them. Now if I come along and delete a pod, it's just a configuration object. So I can delete that configuration object. And now this is where the power of Kubernetes comes in. I've deleted that pod object. And because that pod, that particular pod configuration object has gone away, Kubernetes says, oh, you don't want that anymore. It will actually shut down that, terminate that container, shut down your application and it will go away. But something's going to, magic's going to happen now. You said that you wanted two instances of your application. Now that is the truth. You always want two. You want to delete one pod. So what happens is that even though I delete it and Kubernetes shuts it down, it knows I wanted two. So what happens? It goes and gives me another one back in its place. So I deleted the pod here explicitly. But if this was your application, your application had actually crashed and the whole server had exited and so that container had exited, your application will replace for you automatically. If I just run Docker run command and don't do anything extra and my pod stops, it's gone. It's not there anymore. Kubernetes in this case would replace it for you and keep things running for you. Now, you mentioned for that when Kubernetes is running a cluster of machines. So taking example, if I lose a whole machine now, Kubernetes will notice I've lost a whole machine. All those pods effectively invisible to it now. Again, it will notice this and will start up new instances to your application for you on other nodes in that cluster or automatically. So you're now not going to have a problem with if you've got a very basic system of getting a page at 3a and the morning say your application is down. Kubernetes will worry about keeping things running for you. So we have pods as our application effectively. If I do query on those pods, I can get back and I'll find that each of them has an IP address. When I run Docker on a box by myself and I need multiple instances, that whole host is one IP address. In case of a pod, each pod has a separate IP address. It's like a little host of its own. Even though they may be running on the same underlying host, each has their own IP address. This is very important. Example four about running, having to keep track of instances of your app all running on the same host. If you're doing using Docker, you can't all use port 80. So you have to have this one has to run port 80, this one has to run port 81, this one 82. Because each one has its own IP address, it also has its own port namespace. This means they can all run on port 80. And then don't have a problem having to track all that. Now those IP addresses, if I take that IP address and try to access that from my own medical machine, that will not work. These are IP addresses which are internal to the cluster only. So how do I access it? I got multiple instances and that's a problem because I only want one entry point. And I can't even access them outside the cluster. So just to prove that they do work though, you can actually get into a pod. So one way of getting into pod is very similar to Docker. You can go Docker exec. Same thing with Kubernetes. You can go kubectl exec and get into pod. I can use that IP address but like I said, it's not useful outside the cluster. So what we need to do is, first off, is we need to somehow come up with a single IP address for all of these instances. And that's done using what's called by creating a service resource object. And we can do that using kubectl expose command. So we expose our deployment for our blog. That creates a service object and it's created a new IP address. In this case, we have one IP address. And what it's actually doing under the covers and how this works, it tends to be on how the Kubernetes cluster is set up. But it's setting up an IP router. So I've now got one IP address. And if I contact that IP address, the traffic will go to one of the instances of the pod. So it's handling that load balancing for me across all of those instances. But that IP address is still not accessible outside of the cluster. That IP address, though, will have a host name. Kubernetes has an internal DNS server. So when that is created, that IP address has a host name of blog because that's what my service was created as. So if I need to access that internally, so it was a backend service, like a backend service for your, not a database because that's a bad example because you don't have multiple instances. But if I need to access it, I can just use that as a host name. So I don't have to hard-wire a host name. But to get access to this outside, there are a few different methods you can use. Kubernetes supports the idea of when exposing a service, you can call it, expose it as what's called a node port service. Now that body fault when I created it was what's called a cluster IP. So for node port, it will automatically can create you access to a mass-body cluster. But node port, all that's going to do is give you a magic port number to access. It's not going to be the port number you originally exposed. So it can't be port 80, for example. It's going to be a random port, which is not very useful to us if we want to have a web service because we need them to be on port 80 or 443. The next option is you can use what's called a load balancer service. This one, the underlying infrastructure of Kubernetes, you can have a pool of IP addresses here. When I create a load balancer service, it will allocate me an IP address of my own outside of the cluster. That way, if I use that IP address from outside, it can route traffic through to one of the instances of my pod internally. And that way, I can use a particular port that I want. So you could expose it on port 80. So that's one way. If you're going to use a hosted service for Kubernetes such as Google Cloud, that's what they'll use in that particular case. There's actually a better way. If we're going to use a load balancer service and we have lots and lots of service we need to expose, we have to have an individual IP address for all of them. More useful way of doing it is to have one entry point for the whole cluster and have a router in there, such as HAProxy or Ingenix or even a hardware-based router like F5. And use the fact that when a web browser makes a request with HTTP traffic, it puts a host header on there telling you what host I'm doing there. So the alternative is what we can use is what's called an Ingress in Kubernetes. Now Ingress has started to get a bit more complicated because there isn't a simple way of creating them from the kube-cuttle command line. You have to start delving down into creating YAML descriptions yourself. And this is actually going to become the norm. I started out with using kube-cuttle commands, but in practice, when you start getting anything more complicated and a very simple use case, you're going to have to start learning YAML and JSON and all of these resource descriptions for Kubernetes. You have to start constructing them. That's why your mind starts blowing very quickly with Kubernetes. In the case of Ingress, I can create a YAML description. I need to expose this service with a particular host name, and I want to have all traffic for that routed through to it. And the port internally is going to be AD80, but I'm going to expose this as port 80. So that's all in the YAML file, and I can use kube-cuttle create with that YAML file, pass it in, and it will go and create this Ingress object. Again, it's an object, but the magic in there at Kubernetes is it's going to see that Ingress object created, and it's going to dynamically reconfigure the routing layer for me to do whatever is needed to do to be able to route that traffic through. So it might reconfigure Ingenix or HA proxy or whatever routing system you have, such that now, as long as you have a DNS CNAME outside in the real world which points to the IP address of your router, when someone accessed that host name, your traffic or HTT traffic will get routed through into your application. So they're all bits and pieces. If we step back now and look at all the different things we have got running, we can use kube-cuttle get to query things. Now I mentioned very briefly before about how labels got added onto things. Labels are very important in Kubernetes. It's the way you identify your applications, because you may have lots of different applications in your Kubernetes cluster. If they're all in the same namespace, they're all together, but I need a way of being able to pick out just the ones from application. So what you do is you label them. So I can then start to do queries or operations based on labels. So I'm querying these here based on what's called a label selector, and the label was run, and the name was blog. And I can pull them back out. And it's really weird. You can go kube-cuttle get all, and you'd think, oh, that'll give me back everything in there, which has this label. No, all only actually means a few things. It's really weird. So in this case, ingress is not part of all. So to get them, I had to go query on all the ingress, but I can pull that now based on my label. And I see I've got my deployment, where I wrap the cassette, my pod, my service from when I exposed deployment and my ingress. So lots of different things. But, effectively, they're all just configuration objects, and Kubernetes is all working off them to essentially make that cluster look like what you wanted. Now I mentioned before deleting pods. Now imagine we want to restart our application now. Now we could be brute force and go in there and just delete all our pods using our label selector, so we're only deleting the ones for our application. And it will terminate all those pods, and it will replace them. Great. This is not actually a very good way of deleting or restarting an app, because it's going to delete all the instances of your application at the same time. And there could be a period there where there's no instances of your application running. So people will start getting 500 or five app free service unavailable. Now Kubernetes, unfortunately, doesn't provide a really nice way of doing a restart unless this has changed in your version of Kubernetes. So I did this. So there are some various tricks you can use. Now I mentioned before how Kubernetes watches those configuration objects. And when it sees a change, it will do something to the cluster to make it agree with that. And we can actually trigger a restart by updating a part of what's called the pod template, which is a part of our deployment. In this case, we can update an annotation. We can put the date time on there. And by doing that, there's a change. Kubernetes sees it. Oh, that has changed. I better restart all my application because I've made this change. I'll restart it in case that's significant. It's a bit of a mess. I know there's people who probably know a lot more about aspects of Kubernetes and running it in production than me. They might tell me, oh, no, I'm really stupid. This is actually a really easy way of doing this. That's the best way I know of doing it. Now when we do do it in this way, rather than having a situation where we're deleting all our pods at the same time and having a period of time when application may not be available, what happens is dictated by what's called a deployment strategy. And there's different deployment strategies that you can have in Kubernetes. The default one is what's called a rolling update. So when I do trigger my redeployment using my nasty little technique there, the default is this rolling update. So what will actually happen is rather than delete everything and shut everything down, it will go, OK, you have two instances in my app running already. I'll leave those alone for now, but I will start up a new instance of my application as a new pod. So I've now got three. Once that's up and running properly, it'll shut one of the other ones down. And it'll start a new one, so another second new one. Get that right up and then shut down there. So essentially we're able to do a rolling update. And that is a built-in feature of Kubernetes. You don't have to yourself worry about all the mechanics of doing that. It'll be all handled for you. Now rolling updates are not necessarily always a good idea. They're good for ensuring no uptime redeployment or restart. But if you need to do something like a database migration, using Django, and you've made some changes, you've done your make migrations, you've rebuilt your image, you need to put out that change. And you've set up your image to do a Python managed py migrate as just before the application starts up inside the container. Rolling updates are not necessarily a good idea for that. Because if you're making a change to the schema of your database and you're starting up a new instance of your application while the old one's running, it's going to do that migrate. Meanwhile, your old instance is still handling requests. And you've changed the schema underneath. If the change to that schema is not compatible, then your old instance will start filing requests. So the alternative there is if you did need to have a schema change happening in there, is you can swap to what's called a recreate deployment strategy. And that is where it will shut everything down and then start everything up and new. So that way, if you do doing it, it gives you that option. Now by default, the rolling update was where changing it is a case of having to go in and modify those configs. And there are a few different ways of modifying configs. And I've done it one particular way. It's using kubectl patch that allows you to selectively say I want to make a change inside of an existing config. And I can actually go and change that deployment strategy. You do have to be a little bit careful, though, because some things like deployment strategy, for example, you can have a type which is recreate. I want to set that stat. But I may have to remove bits and pieces which are in there from the existing rolling update. In this case, there is a rolling update parameters part of the configuration. If I leave that in there and just change it to create, it will have probably, I'll get an error. It won't actually allow me to do it because I've got a thing in there which is not valid for that. So you do a little bit tricky. But you can do this with patch. And I mentioned patch only because I can show it easily on slide. But if you're starting to try and script these things, you do need an easy way of scripting. You don't want to be able to necessarily jump in and edit raw configurations yourself. But these are the ways you can edit things. There's Kupedal patch, which is that one example of it. You can use Kupedal edit. And what it will do is actually just throw you into an editor with the raw YAML. And you can start making changes. You save it. And it will go up and just replace the existing config. And I'll do that. Other options, and this is more typical option you'll end up using, more complicated ones, is you've created resources using Kupedal create from original raw file. If you want to make changes, you're going to use replace or apply to essentially have a copy of your original config. You'll make the changes and you'll apply that then to replace the whole lot in one go. Preferably you'll have your config under version control so that you can recreate it later and so on. So avoid patch and edit. It would be great for doing ad hoc, mucking around and so on, but not good if you want a ability to reproduce things. Docs will tell you about all the different attributes. You can change your things. There's actually a Kupedal explain command where you can actually get very simple docs out very quickly on resources if you want to need to jump into things. Now deployment strategy, I mentioned rolling update. When you start up that new instance of the application, how though do you know that is ready to handle requests? What if there's a failure with that starting up and it's not ready to accept requests? You can add what's in called health checks. First one is a readiness probe. So what happens, you can define a readiness probe on there and you can define it in different ways. The typical ways is just use your HTTP get. And as that, when that application is just started up, it'll start triggering that probe and when it gets back a successful response that it has worked, it'll go, good, that one's fine. I can add that into my service. There's a list of end points that the service maps to and shut down my old one. So health checks are very important. You're being able to do things like rolling updates nicely such that you know your application is going to be ready. Liveness probe is another one, which is a thing that's actually going to always run as well, which can tell you whether your application is actually working properly. If either of those fails, readiness probe fails, it can take that particular instance out of the service so it's no longer being traffic routed to it and live in this probe fails, it will actually shut the pod down and replace it. So I can create that. I can again just patch it in. In this case of working out where in the config I need to do it. And I said you can also, instead of HTTP get, you can also use for certain things socket connections or run it to hand-to-hand inside a container to implement those probes. With database grace regulations, part of that deployment strategy, you can also define what are called deployment hooks. So in the case of a rolling deployment, when I trigger that deployment, I can have what's called a pre-hook. Before anything happens, I can define something. When it's finished and it's done all the replacements, I can have another hook which is I can have a post hook. For database migrations, our Python migrate, Python managed to power Managra. Some embedding into the container itself such as run when the container starts, which would get run every, for every single container if you have more than one. An alternative there is to break that out and if you're using recreate deployment strategy, you can define what's deployment hook. So you can actually run up a container which just is going to do the migration. So in this case what would happen is you have all the existing apps running, it would shut them all down, you can run a container to do the migration and then you can actually start up all the new ones. So deployment strategy. All these things are in part gubernators as default thing and you don't have to build them in. You can build on your top of this and build your own custom deployment strategies. You can do blue-gray deployments. So blue-gray deployments, you might, for example, have one deployment. You might decide, okay, I'll leave that one alone. I'll create a totally new deployment. Get all that up and running and test it. And then all I need to do with blue-gray strategy is essentially switch the ingress to point at my alternative deployment. So you can build up these alternative deployment strategies as well as if you want. Canary, A-B testing and various other ones like that. Next one is how do I update my application? Now I cheated. When I cheated originally I said, Kube pedal run, I said run up. My image, I've been used the latest version. That's actually really bad. Never do that. Always use a version number because latest can map to anything. Every time I do a Docker build and I do a Docker tag, latest effectively can map to that. So it can change under me. So if I did want to upgrade my application now, I have one dot zero version running. I can just change the image name in the deployment config and change the new tag and it will do a redeployment for me. All automatically for you. Storage, I'm running out of time. So storage, people originally with Docker saw it as being for 12 factor apps or cloud native apps that didn't have any storage. If you're using Heroku, you're very, very familiar with that. If you deploy stuff to Heroku, there is no storage. If I write to the local file system application, I've lost that. You always have to have data outside in a separate database or storing stuff in S3 or other external. One of the good things about Kubernetes is it supports the concept of persistent volumes. This is very handy if you need to bring in apps, so-called legacy apps or non-12 factor apps which do have a requirement for storage or if you need to just support storing of image uploads in a blog app, for example. So I can actually mount in persistent volumes into my container. The way this works is that I send you to tell Kubernetes, I need persistent storage, I need this amount, and this is the type of storage I need. This is very important. So there's different types of storage. Read write ones, read only many, and read might many. Now read write ones, if you're familiar with Amazon, that equates to elastic block storage. Elastic block storage volumes can only be mounted on one node at a time. Read write many down the bottom, there's traditional files in storage like NFS. You can mount that wherever you want. And this is important because if I have an app and I want to scale it up and have multiple instances, then I can't use the top one because Kubernetes is going to make decisions about where it runs the instances of my apps and it could run multiple instances across different nodes. So if you're using the wrong storage type and you say read write ones, it'll only bring up one instance or whatever instance it puts on that node. And this is the really thing about Kubernetes you have to be careful of. If you can set up a configuration and that configuration could be wrong or be telling Kubernetes to do something that it can't satisfy. And when I run that command or update the config, you do not get an error. Kubernetes will just spin it's wheels because it can't actually satisfy it. That's one of the weird things. And in storage, an example, if I use read write one storage, I think it would be mounted on one node, 10 instances, why are they only at four instances running? It's because Kubernetes couldn't satisfy that request because it couldn't perhaps put more instances to wrap on that one node. It's trying to distribute them elsewhere and it's just sitting there waiting for storage which you can't get. So if you do, I start with using EBS on Amazon, you are stuck with one replica or you have to do tricks like tell Kubernetes to set up all my instances, my app to run on one node. But then you lose your fault tolerance a bit because if I lose the whole node, I've got no instances because the idea of Kubernetes will distribute the instances across the nodes. So if you do lose a node, then it's still got other instances running. Again, persistent volumes is like Ingress. You have to start marking around with raw definitions and then you can just create it, you have to start marking around the deployment config, you can change the deployment config and say, okay, I need this persistent volume claim for this deployment but I also need to then say where that is going to be mounted inside of that particular container. So there's lots of marking around to bring all these things together. But then once we do have storage, we want to get in there and do things like set up our super user cell, we can do that. We can go in there, kubect, run our create super user. I'm using SQLite and fine, I've got my storage. If I want to change config, like various systems like this, I can set environment variables which pass down into configs. So if I'm using modWiz, I can make that deployment config or redo. More importantly, environment passing is good for linking up things like databases. So I could have just passed in all the details of where my database is and environment variables, but Kubernetes offers another thing called secrets and that is the ability to put a configuration in what's called a secret and I can actually have that as a separate object and I can use that from multiple things. So here I've already pre-deployed my database, it has a secret with my credentials in it and without needing to know what they are, I can say, okay, set my environment but pull those variables from the secret and set them all automatically and I can link them together. In addition to secrets, another thing called config maps, they essentially work exactly the same but secrets have some better guarantees on how that information is managed such that it's not stored on disk. So if I shut down a machine, ship it out, I haven't put my secrets out and given them away to someone. And I can get back and view my configuration, you can see where it's coming from. Now that's the guts of it, that's a very simple example of running an app. Now I mentioned these distributions earlier, distributions can make things simpler because they can add on extra bits which make it easier from a developer's perspective. Kubernetes is very much regarded as being an operations platform. As developers, that's not great, operations tend to, oh, that's our stuff, we don't want to touch it. That's what you might look at some of the distributions, so OpenShift from Red Hat and OKD which is the upstream version of that and also Pivotal, we're coming over from CloudFabric, they have mechanisms or tooling which are much more developer friendly and allow you to build up things. Now I'm going to skip through a quick example of how to do what I just did but using OpenShift using their tooling. It's still Kubernetes underneath. There's a few little things that are different but I can go into OpenShift and one of the things here is that in what I did before I had an existing image, here I'm going to go into OpenShift and say, here is my source code repository on GitHub, deploy it for me and it's going to do that platform as a service type thing that you've made with Heroku. It'll take my source code and build into an image for me and it will go off and create a deployment for me, create a service and get it all running for me. I can also do things like adding in volume very easily because it has extra commands in there to very easily claim storage and mount it in straight away. So I've deployed it, got my service already, it didn't have to expose it and I can add in my storage and I can then expose it. So three steps, I've very quickly done all of what I did before. In this case, some of the things underneath, if I now query it back, you'll see a lot of the things are a bit different and this is a bit of a historical because when OpenShift was first created, things like deployment didn't exist in Kubernetes, things like ingress didn't exist in Kubernetes. So you might see people talking about, well OpenShift's not really Kubernetes because it's different. It's just a history of it. Things like these didn't exist, so OpenShift went and created their own and that's actually fed into then Kubernetes in developing things like deployment and ingress in Kubernetes. OpenShift is in the process of migrating across to using these new ones in there. But do look at some of the distributions, they do provide simpler tooling for this. And one further example in OpenShift is that if I want to deploy a database, there is, in OpenShift has a concept of templates in there and I can say here, create me database and that's all I need to do. It'll go and create it, create me credentials, create that secret which I can then bring it from environment into map, just like that. There's also have web consoles which are much more developer friendly. A default plain vanilla communities doesn't really have a console. There is a management console you can deploy yourself. All that stuff comes out of the box with OpenShift. And it's here, I'm running out of time. So cleaning up, I've mentioned labels before, it's been very important. And that's where it comes in, important, deleting stuff. You use your labels for deleting things. I mentioned templates. There are other templing systems too to look at, Helm and Ksonnet. There are others as well. They help you to essentially define your application as a whole lot of templates and they to vary easily to reproducible deployments. So look at those. Building, you may be familiar with Docker builds because that all becomes standards under the OpenCotange initiative. There are now alternative tools for doing builds. Builder is an example of doing build. Has a companion program called Scopio for doing pushing images and so on. And they're all right together with thing called Podman. The source to image is what OpenShift uses for doing its past type builds. Very quick one, don't run up, don't design your app images though to run as root. Please don't do that. Run them so they can run as arbitrary ideas, user ideas much better. And there's various tricks. I'll skip this because I'm running out of time. You can go back. If you want to play with it, some resource we can go. Kubernetes by example is a great place we can go to learn about some of the low level concepts in Kubernetes, what they are for, how to use them. There's a site called Catechord where you can go and it will spin up an instance of Kubernetes for you. And it will have on one side a set of instructions, a terminal, and you can go through and just do it all on the spot. Just click on the instructions and it will run them all for you. You can play with Kubernetes that way. If you want to run Kubernetes yourself on your own box, it's Minikube. So rather than go and try and do the whole hog of setting up a whole cluster, you can use Minikube. It just runs up inside of VM. And there are similar things for OpenShift. It has Catechord environment as well, which is learnopenshift.com. It has Minikshift, same as Minikube. And finally, if you want resources, I've got my name on a couple of books if you want to look at this from OpenShift side more. And they are free downloads. And that's it. I don't think we've got any time for questions. So I'll be over. You can get me later. No time for questions. If Graham's willing to take questions in the hall afterwards, you can find them then. So thank you, Graham. Thank you.
 If you're not familiar with who I am, I am not a Django developer.  I know about the internals of Django, I haven't had a deploy it, but I don't actually have  really had a right Django apps.  So my history is I am the developer of modWizki for Apache, for hosting applications with  Apache, also a response for a Python package called wrapped, which is for implementing  decorators, and you may also know of me from Nurellic, where I wrote the Python agent for  Nurellic and do lots of evil things to your Python code.  I don't work at Nurellic now, I currently work at OpenShift, Red Hat, where I am a developer  advocate for Kubernetes and OpenShift.  And I always have problems with my pointer.  We'll go to backup plan.  That's better.  So Kubernetes, I don't know how many here are familiar with Kubernetes.  Anyone want to put their hands up and know about it?  So for those who have got into Kubernetes already, would this be a fair explanation,  would you say?  Yes, more tears.  So Kubernetes, it's an up and coming thing, but it's a very large ecosystem and can be  very, very, very confusing when you start to get in.  It's a very large learning curve.  If we look at the landscape of what exists around Kubernetes, there is a lot and a lot  of stuff now starting to come around it.  They are software packages you can use with Kubernetes, but also all the different companies  are bringing stuff to it.  So there's a lot in there which potentially could learn.  So it can, depending on how deep you go into that, you can get very, very complicated and  it's very, very mind-blowing.  So if you are getting into it, be patient because it can take a long time to get up  to speed on it.  To make it easier rather than just dive in and think, okay, I'll go and get original  Kubernetes distribution and start using that, I strongly encourage you to look around at  easier ways of getting into Kubernetes.  And there's different ways there.  We've hosted versions of Kubernetes and also distributions of Kubernetes.  The way to think about Kubernetes, it's like a kernel.  So whereas you've got a Linux kernel, which is the guts in there, which helps you to run  up a patient on a Linux operating system, you're never going to install that kernel  yourself.  It's always better to go get a distribution.  So you're always going to get a Ubuntu or Fedora or CentOS, Red Hat or other Linux distribution.  So it's always better to go through that because they will provide you an experience which  is going to provide extra bits on top of Kubernetes that's going to make your job easier to understand  and use it.  So think of Kubernetes as a bit like a kernel.  And in this case, it's going to be the kernel for a cluster of machines where you're going  to want to deploy an application.  I'll say more about that later.  So stepping back from that bit of warning about getting into Kubernetes and where you  should perhaps go to start looking at it, what exactly are we talking about?  We are talking about containers.  We all heard of the containers now.  It's the big buzzword from the last few years.  This has actually been around for quite a long time.  Google in their infrastructure has been using containers since about 2002, 2003.  I can't remember exactly when.  But even before that, it was the similar sorts of ideas of what you were doing, talking about  containers existed in the form of Chirrut Jails in BSD and also Solaris Zones.  So they're not a new concept.  But containers are a thing that has come more popular of late, and it became popular  because of a company which, when they announced it, was called DotCloud, but you'll know them  as Docker.  So this company called DotCloud at PyCon at Santa Clara in a lightning talk showed a really  simple way of running up an application inside of what we call a container.  So what is a container?  This way to think about it is to compare in terms of what we've done in the past.  If you're familiar with virtualization and the idea of running an operating system inside  of Sandbox on your operating system, so I might have a Mac and I want to run Windows  or Linux, I can install a version of an operating system, Windows or Linux, inside a virtual  environment or virtual machine on my box, in a little Sandbox.  And I'm quite happy to work away in that.  And I can actually install multiple operating systems, multiple systems of Windows or a mix  of Windows and Linux.  And then inside of that, I can run an application.  And it's providing a Sandbox, an environment where you can run stuff, and it's kept away  and it's isolated from your host operating system, but also your other operating system  running in a separate virtual environment, virtual machine.  So containers, the idea behind them is a very lightweight way of running up an application  inside of a container or Sandbox on a host operating system.  Now the difference from traditional virtual machines is that in a traditional virtual  machine, you are running a whole operating system inside that Sandbox.  And then you're running your apps.  So you've got a whole lot of application services in your system services and therefore that  operating system, as well as your application.  In a container, you are making use of the fact you've got a host operating system running  under or underneath already, and you're going to run just the processes for your application  in that Sandbox.  So you're not going to run additional processes in there for system services.  It's just the processes for your application.  So with Django conference, so in this case, we're talking about Django web application.  So you might be running your Django application under modWizGi, which I hope you are, but  I know you're not.  You're probably using UWizGi or Goonicorn, I'm sure.  So all you're doing is you're running that WizGi server inside of that container with  your Django application in it.  But because it's in this containerized environment, it's still protected.  If I get into that container, all I can see is those processes for that application.  I cannot see operating system services in the underlying host operating system, and I can't  see applications inside of other containers.  So it's protecting things.  So that's what it is.  And it's meant to be this more lightweight thing, because we're not starting up a whole  operating system.  It's much quicker.  You only need to start up your just your processes.  Now to run off something in a container, what Docker did in terms of what they announced  was they provided a couple of things.  And it's all focused around tooling.  A container, as I said, and not a new thing.  They've existed for a long time.  And the way you use them normally is you're using what's called Linux containers, or LACC.  So there is this way of running up processes in constrained environments on the operating  system.  But that was always a difficult thing.  Docker made it quite simple because they provided this program called Docker.  And there was two important things for that.  There was one, which was Docker build command.  And the idea there is you could provide what was called a Docker file with a set of instructions  of how to build an image which contained all of the bits and pieces you need.  So you start out with a base image for the operating system.  So it might be a 1, 2.  And then you'll add your instructions on top of how to actually install your application  with all the dependencies it needs, all the different Python packages, into that.  And they all get built into an image.  You want to then run that image, and we can run that up on our box.  But what usually is going to happen is we're going to push that image up into what's called  an image registry.  Once you've got an image registry, we can then do Docker run, and we run it up.  And away we go.  So that's what they made a very simple process to do.  Before that, using LACC was a lot of different steps to do all of that.  The Docker run command, when we take that image, is going to do a lot of work to create  that sandbox environment for you.  But ultimately, even though we talk about Docker, it's still underneath using Linux  containers.  It's using the features of the operating system.  Docker was just bringing the tooling to make that easy to do.  Now running single containers with your application, we can do that.  You can get it working.  A bit of work to do, you can be happy.  The problems occur when we start to want to run many, many containers, especially if we  want to run across multiple hosts.  Because we can only scale up applications, the number of instances or replicas of that  application we were running on the single host so far.  We're going to run out of resources.  So we need to start to put them across hosts.  Now Docker was great for running up an application in container on one machine.  When we start to go beyond that one machine, that's when it starts to get more difficult.  You have to start tracking.  I need to run 10 instances.  I can only fit five on this host.  I have to run five here.  I have to remember I've got five here.  I have to run five over here.  But then I've got all these different instances of them and they all are going to listen.  They all expect to listen on port 80 for HTTP traffic.  They can't all listen to port 80 because they're on the same host.  So we need to worry about port assignments and put them all on different ports.  But then you have to worry about, well, how do I distribute traffic?  You have to start putting load balances across and distribute traffic.  So very, very quickly it starts to become very complicated to be able to do that all  yourself and track it.  And this is where Kubernetes comes in.  Kubernetes, as my kernel, as I mentioned before, is a means of managing the deployment  of applications inside containers distributed across a cluster of machines.  So it's a typical Kubernetes cluster.  What you'll have is some master nodes.  I've got one here.  But if we talk about fault tolerant or high availability systems, you'll actually have  three at least, an odd number.  And you may have other infrastructure nodes in there which have other bits and pieces  like the router.  But I've simplified this.  I've got one master node and I'm saying everything's running on there.  That's the smart, main smarts of Kubernetes.  That's where the database is there, which is tracking how, what we're deploying and  so on.  And also router and other bits and pieces.  And then we all have our nodes.  That's where your applications are going to run.  So that's what Kubernetes is going to do, is provide that ability.  Now, in this talk, I'm not going to delve down into the depths of how Kubernetes works.  I'm just going to step through running a, deploying an application inside of Kubernetes  using Kubernetes command line tool.  And I'm going to use that as a means to just introduce different concepts of Kubernetes  so that if you are going into Kubernetes, at least then you've heard what these concepts  are once before at least, and they might make a bit more sense to you.  So deploying.  We've already built our image with Docker build, for example.  We have it already up in an image registry somewhere.  And a popular place that people do store images up, a place called Docker Hub.  That was a public image registry which was run originally created by the Docker company,  Inc.  But there are other image registries you can run your own as well.  So if I want to run up an image inside of Kubernetes, so if you're using Docker, I'd  use Docker run.  Here I'm just going to use kubectl run.  It's very, very similar.  I want to name that.  So I'm going to name my application blog.  I tell it where my image is and I tell it the port that I need to expose.  If you've used Docker before, you'll see it's very, very similar to Docker run.  The difference is that if I use Docker run, it's going to run it on my own machine, where  the Docker service is, or the one you've logged into.  Here, because we've got a cluster of machines, and I've just said, okay, I want to run run  instance, Kubernetes is going to go, okay, I've got a cluster of machines I can run this  on.  It will go and work out for us where to run that and make a decision for us and get it  up and running.  When we do that, when we run Docker run, it's actually taking an action of doing that.  It's actually going to essentially go right through and run it up and start it.  Kubernetes works a bit different to how Docker is.  When I run that Docker run command, it's not actually really starting things up.  What it's actually doing is going to set up some configuration.  If we actually run that same command and put dry run minus output YAML file in there, you  can see what it's doing.  What all it's actually going to do is create a YAML description of what it is I want deployed.  That's actually going to be stuck inside of what's called a data store inside of Kubernetes,  which actually uses an XED.  That's actually all it's going to do.  That's all the command does.  Updates that database of information.  Where Kubernetes works is what's called a declarative model.  Rather than actually you do something that causes an action immediately, all you're doing  is updating your configuration of what I want my application deployment to look like.  Kubernetes comes along and is always checking these configuration objects, which exists  inside of the database and goes, oh, I've got a new configuration object, or a configuration  object has changed.  I need to now go make the cluster or what you want for your deployment, what you have  in the cluster, agree with what that configuration says.  Because it's a new one, Kubernetes goes, good, oh, new deployment configuration, I'll go  and do the steps needed to actually satisfy what your configuration says.  In this case, our deployment, it was given a name.  It was given some labels automatically to help us identify our configuration later on,  which is very important.  It had a number of instances, which was just one initially.  We have a description of container, of what, and our image, which we're going to run inside  of that container.  So Kubernetes will go off and run up an instance of that for me.  I've told it, I only want one, so it'll go and do one.  I can then query back once I've done that.  I can do a query back of the resources which we've created.  You'll actually see my deployment there, you'll actually see some other resources created  there as well.  This is where my command only created one, the deployment.  But Kubernetes has stepped in and said, OK, to satisfy your requirement of having your  deployment, well, I actually need to go and create some other resource objects in there,  more configuration.  Now, the first one is what's called a replica set.  It's the thing which is going to manage your instances of replication and so.  So it pulls out of that deployment config information about the number of replicas you  needed, the description of the container and the image that you needed to run.  It's created a replica set for me.  There's an aspect of Kubernetes which sits in there and looks for now for replica sets.  When it sees that's created, it goes, oh, I need to now do something based on this replica  set.  So it pulls out the description of the container, it knows only one one instance, and it creates  what's called a pod.  Now, pod, I mentioned containers.  Well, what's the relationship?  Normally, when I say container, pod is the same thing.  But in effect, a pod can be one or more containers.  So when we talked about our description back there, you can see our containers in our demo  description is actually a list.  But I'm only describing one.  So technically, I can have more than one container in what's called a pod.  This name pod, where does it come from?  Pod of Wales.  And Docker, well, they had their logo of a whale.  So that's where the naming come from.  But you can more or less think about a pod as being a container most of the time.  So now we've introduced a deployment.  That triggers the creation of a replica set.  And I've got a pod, which is my running instance of my container.  If I need to look at what my application is doing, I can use that pod as the name of that  pod as a means of getting out, say, logs and find out what happened when my application  started up.  But more importantly is if I want to scale it, what happens then?  If I want to scale it, there's a command called kubectl scale.  And I can actually refer to my deployment config for that.  And I say, I now you want to have two replicas of my application.  And all again, this has gone back.  All this has gone back has changed that deployment config.  Kubernetes has noticed a change in the configuration.  It says, I need to now make the cluster match that.  So it will go and update the replica set.  Say, I've got two replicas.  And then the controller for that, looking at replicas, kicks in and goes, oh, two, I'll  create another pod for you.  And Kubernetes is once again making decisions for you about where it's going to run them.  Now if I come along and delete a pod, it's just a configuration object.  So I can delete that configuration object.  And now this is where the power of Kubernetes comes in.  I've deleted that pod object.  And because that pod, that particular pod configuration object has gone away, Kubernetes  says, oh, you don't want that anymore.  It will actually shut down that, terminate that container, shut down your application  and it will go away.  But something's going to, magic's going to happen now.  You said that you wanted two instances of your application.  Now that is the truth.  You always want two.  You want to delete one pod.  So what happens is that even though I delete it and Kubernetes shuts it down, it knows  I wanted two.  So what happens?  It goes and gives me another one back in its place.  So I deleted the pod here explicitly.  But if this was your application, your application had actually crashed and the whole server had  exited and so that container had exited, your application will replace for you automatically.  If I just run Docker run command and don't do anything extra and my pod stops, it's gone.  It's not there anymore.  Kubernetes in this case would replace it for you and keep things running for you.  Now, you mentioned for that when Kubernetes is running a cluster of machines.  So taking example, if I lose a whole machine now, Kubernetes will notice I've lost a whole  machine.  All those pods effectively invisible to it now.  Again, it will notice this and will start up new instances to your application for you  on other nodes in that cluster or automatically.  So you're now not going to have a problem with if you've got a very basic system of getting  a page at 3a and the morning say your application is down.  Kubernetes will worry about keeping things running for you.  So we have pods as our application effectively.  If I do query on those pods, I can get back and I'll find that each of them has an IP  address.  When I run Docker on a box by myself and I need multiple instances, that whole host is  one IP address.  In case of a pod, each pod has a separate IP address.  It's like a little host of its own.  Even though they may be running on the same underlying host, each has their own IP address.  This is very important.  Example four about running, having to keep track of instances of your app all running  on the same host.  If you're doing using Docker, you can't all use port 80.  So you have to have this one has to run port 80, this one has to run port 81, this one  82.  Because each one has its own IP address, it also has its own port namespace.  This means they can all run on port 80.  And then don't have a problem having to track all that.  Now those IP addresses, if I take that IP address and try to access that from my own  medical machine, that will not work.  These are IP addresses which are internal to the cluster only.  So how do I access it?  I got multiple instances and that's a problem because I only want one entry point.  And I can't even access them outside the cluster.  So just to prove that they do work though, you can actually get into a pod.  So one way of getting into pod is very similar to Docker.  You can go Docker exec.  Same thing with Kubernetes.  You can go kubectl exec and get into pod.  I can use that IP address but like I said, it's not useful outside the cluster.  So what we need to do is, first off, is we need to somehow come up with a single IP address  for all of these instances.  And that's done using what's called by creating a service resource object.  And we can do that using kubectl expose command.  So we expose our deployment for our blog.  That creates a service object and it's created a new IP address.  In this case, we have one IP address.  And what it's actually doing under the covers and how this works, it tends to be on how  the Kubernetes cluster is set up.  But it's setting up an IP router.  So I've now got one IP address.  And if I contact that IP address, the traffic will go to one of the instances of the pod.  So it's handling that load balancing for me across all of those instances.  But that IP address is still not accessible outside of the cluster.  That IP address, though, will have a host name.  Kubernetes has an internal DNS server.  So when that is created, that IP address has a host name of blog because that's what my  service was created as.  So if I need to access that internally, so it was a backend service, like a backend service  for your, not a database because that's a bad example because you don't have multiple  instances.  But if I need to access it, I can just use that as a host name.  So I don't have to hard-wire a host name.  But to get access to this outside, there are a few different methods you can use.  Kubernetes supports the idea of when exposing a service, you can call it, expose it as what's  called a node port service.  Now that body fault when I created it was what's called a cluster IP.  So for node port, it will automatically can create you access to a mass-body cluster.  But node port, all that's going to do is give you a magic port number to access.  It's not going to be the port number you originally exposed.  So it can't be port 80, for example.  It's going to be a random port, which is not very useful to us if we want to have a web  service because we need them to be on port 80 or 443.  The next option is you can use what's called a load balancer service.  This one, the underlying infrastructure of Kubernetes, you can have a pool of IP addresses  here.  When I create a load balancer service, it will allocate me an IP address of my own outside  of the cluster.  That way, if I use that IP address from outside, it can route traffic through to one of the  instances of my pod internally.  And that way, I can use a particular port that I want.  So you could expose it on port 80.  So that's one way.  If you're going to use a hosted service for Kubernetes such as Google Cloud, that's what  they'll use in that particular case.  There's actually a better way.  If we're going to use a load balancer service and we have lots and lots of service we need  to expose, we have to have an individual IP address for all of them.  More useful way of doing it is to have one entry point for the whole cluster and have  a router in there, such as HAProxy or Ingenix or even a hardware-based router like F5.  And use the fact that when a web browser makes a request with HTTP traffic, it puts a host  header on there telling you what host I'm doing there.  So the alternative is what we can use is what's called an Ingress in Kubernetes.  Now Ingress has started to get a bit more complicated because there isn't a simple way of creating  them from the kube-cuttle command line.  You have to start delving down into creating YAML descriptions yourself.  And this is actually going to become the norm.  I started out with using kube-cuttle commands, but in practice, when you start getting anything  more complicated and a very simple use case, you're going to have to start learning YAML  and JSON and all of these resource descriptions for Kubernetes.  You have to start constructing them.  That's why your mind starts blowing very quickly with Kubernetes.  In the case of Ingress, I can create a YAML description.  I need to expose this service with a particular host name, and I want to have all traffic  for that routed through to it.  And the port internally is going to be AD80, but I'm going to expose this as port 80.  So that's all in the YAML file, and I can use kube-cuttle create with that YAML file,  pass it in, and it will go and create this Ingress object.  Again, it's an object, but the magic in there at Kubernetes is it's going to see that Ingress  object created, and it's going to dynamically reconfigure the routing layer for me to do  whatever is needed to do to be able to route that traffic through.  So it might reconfigure Ingenix or HA proxy or whatever routing system you have, such  that now, as long as you have a DNS CNAME outside in the real world which points to  the IP address of your router, when someone accessed that host name, your traffic or HTT  traffic will get routed through into your application.  So they're all bits and pieces.  If we step back now and look at all the different things we have got running, we can use kube-cuttle  get to query things.  Now I mentioned very briefly before about how labels got added onto things.  Labels are very important in Kubernetes.  It's the way you identify your applications, because you may have lots of different applications  in your Kubernetes cluster.  If they're all in the same namespace, they're all together, but I need a way of being able  to pick out just the ones from application.  So what you do is you label them.  So I can then start to do queries or operations based on labels.  So I'm querying these here based on what's called a label selector, and the label was  run, and the name was blog.  And I can pull them back out.  And it's really weird.  You can go kube-cuttle get all, and you'd think, oh, that'll give me back everything  in there, which has this label.  No, all only actually means a few things.  It's really weird.  So in this case, ingress is not part of all.  So to get them, I had to go query on all the ingress, but I can pull that now based on  my label.  And I see I've got my deployment, where I wrap the cassette, my pod, my service from  when I exposed deployment and my ingress.  So lots of different things.  But, effectively, they're all just configuration objects, and Kubernetes is all working off  them to essentially make that cluster look like what you wanted.  Now I mentioned before deleting pods.  Now imagine we want to restart our application now.  Now we could be brute force and go in there and just delete all our pods using our label  selector, so we're only deleting the ones for our application.  And it will terminate all those pods, and it will replace them.  Great.  This is not actually a very good way of deleting or restarting an app, because it's going to  delete all the instances of your application at the same time.  And there could be a period there where there's no instances of your application running.  So people will start getting 500 or five app free service unavailable.  Now Kubernetes, unfortunately, doesn't provide a really nice way of doing a restart unless  this has changed in your version of Kubernetes.  So I did this.  So there are some various tricks you can use.  Now I mentioned before how Kubernetes watches those configuration objects.  And when it sees a change, it will do something to the cluster to make it agree with that.  And we can actually trigger a restart by updating a part of what's called the pod template, which  is a part of our deployment.  In this case, we can update an annotation.  We can put the date time on there.  And by doing that, there's a change.  Kubernetes sees it.  Oh, that has changed.  I better restart all my application because I've made this change.  I'll restart it in case that's significant.  It's a bit of a mess.  I know there's people who probably know a lot more about aspects of Kubernetes and running  it in production than me.  They might tell me, oh, no, I'm really stupid.  This is actually a really easy way of doing this.  That's the best way I know of doing it.  Now when we do do it in this way, rather than having a situation where we're deleting all  our pods at the same time and having a period of time when application may not be available,  what happens is dictated by what's called a deployment strategy.  And there's different deployment strategies that you can have in Kubernetes.  The default one is what's called a rolling update.  So when I do trigger my redeployment using my nasty little technique there, the default  is this rolling update.  So what will actually happen is rather than delete everything and shut everything down,  it will go, OK, you have two instances in my app running already.  I'll leave those alone for now, but I will start up a new instance of my application  as a new pod.  So I've now got three.  Once that's up and running properly, it'll shut one of the other ones down.  And it'll start a new one, so another second new one.  Get that right up and then shut down there.  So essentially we're able to do a rolling update.  And that is a built-in feature of Kubernetes.  You don't have to yourself worry about all the mechanics of doing that.  It'll be all handled for you.  Now rolling updates are not necessarily always a good idea.  They're good for ensuring no uptime redeployment or restart.  But if you need to do something like a database migration, using Django, and you've made some  changes, you've done your make migrations, you've rebuilt your image, you need to put  out that change.  And you've set up your image to do a Python managed py migrate as just before the application  starts up inside the container.  Rolling updates are not necessarily a good idea for that.  Because if you're making a change to the schema of your database and you're starting up a  new instance of your application while the old one's running, it's going to do that migrate.  Meanwhile, your old instance is still handling requests.  And you've changed the schema underneath.  If the change to that schema is not compatible, then your old instance will start filing requests.  So the alternative there is if you did need to have a schema change happening in there,  is you can swap to what's called a recreate deployment strategy.  And that is where it will shut everything down and then start everything up and new.  So that way, if you do doing it, it gives you that option.  Now by default, the rolling update was where changing it is a case of having to go in and  modify those configs.  And there are a few different ways of modifying configs.  And I've done it one particular way.  It's using kubectl patch that allows you to selectively say I want to make a change inside  of an existing config.  And I can actually go and change that deployment strategy.  You do have to be a little bit careful, though, because some things like deployment strategy,  for example, you can have a type which is recreate.  I want to set that stat.  But I may have to remove bits and pieces which are in there from the existing rolling update.  In this case, there is a rolling update parameters part of the configuration.  If I leave that in there and just change it to create, it will have probably, I'll get  an error.  It won't actually allow me to do it because I've got a thing in there which is not valid  for that.  So you do a little bit tricky.  But you can do this with patch.  And I mentioned patch only because I can show it easily on slide.  But if you're starting to try and script these things, you do need an easy way of scripting.  You don't want to be able to necessarily jump in and edit raw configurations yourself.  But these are the ways you can edit things.  There's Kupedal patch, which is that one example of it.  You can use Kupedal edit.  And what it will do is actually just throw you into an editor with the raw YAML.  And you can start making changes.  You save it.  And it will go up and just replace the existing config.  And I'll do that.  Other options, and this is more typical option you'll end up using, more complicated ones,  is you've created resources using Kupedal create from original raw file.  If you want to make changes, you're going to use replace or apply to essentially have  a copy of your original config.  You'll make the changes and you'll apply that then to replace the whole lot in one go.  Preferably you'll have your config under version control so that you can recreate it later  and so on.  So avoid patch and edit.  It would be great for doing ad hoc, mucking around and so on, but not good if you want  a ability to reproduce things.  Docs will tell you about all the different attributes.  You can change your things.  There's actually a Kupedal explain command where you can actually get very simple docs  out very quickly on resources if you want to need to jump into things.  Now deployment strategy, I mentioned rolling update.  When you start up that new instance of the application, how though do you know that is  ready to handle requests?  What if there's a failure with that starting up and it's not ready to accept requests?  You can add what's in called health checks.  First one is a readiness probe.  So what happens, you can define a readiness probe on there and you can define it in different  ways.  The typical ways is just use your HTTP get.  And as that, when that application is just started up, it'll start triggering that probe  and when it gets back a successful response that it has worked, it'll go, good, that one's  fine.  I can add that into my service.  There's a list of end points that the service maps to and shut down my old one.  So health checks are very important.  You're being able to do things like rolling updates nicely such that you know your application  is going to be ready.  Liveness probe is another one, which is a thing that's actually going to always run  as well, which can tell you whether your application is actually working properly.  If either of those fails, readiness probe fails, it can take that particular instance  out of the service so it's no longer being traffic routed to it and live in this probe  fails, it will actually shut the pod down and replace it.  So I can create that.  I can again just patch it in.  In this case of working out where in the config I need to do it.  And I said you can also, instead of HTTP get, you can also use for certain things socket  connections or run it to hand-to-hand inside a container to implement those probes.  With database grace regulations, part of that deployment strategy, you can also define what  are called deployment hooks.  So in the case of a rolling deployment, when I trigger that deployment, I can have what's  called a pre-hook.  Before anything happens, I can define something.  When it's finished and it's done all the replacements, I can have another hook which is I can have  a post hook.  For database migrations, our Python migrate, Python managed to power Managra.  Some embedding into the container itself such as run when the container starts, which  would get run every, for every single container if you have more than one.  An alternative there is to break that out and if you're using recreate deployment strategy,  you can define what's deployment hook.  So you can actually run up a container which just is going to do the migration.  So in this case what would happen is you have all the existing apps running, it would shut  them all down, you can run a container to do the migration and then you can actually  start up all the new ones.  So deployment strategy.  All these things are in part gubernators as default thing and you don't have to build  them in.  You can build on your top of this and build your own custom deployment strategies.  You can do blue-gray deployments.  So blue-gray deployments, you might, for example, have one deployment.  You might decide, okay, I'll leave that one alone.  I'll create a totally new deployment.  Get all that up and running and test it.  And then all I need to do with blue-gray strategy is essentially switch the ingress to point  at my alternative deployment.  So you can build up these alternative deployment strategies as well as if you want.  Canary, A-B testing and various other ones like that.  Next one is how do I update my application?  Now I cheated.  When I cheated originally I said, Kube pedal run, I said run up.  My image, I've been used the latest version.  That's actually really bad.  Never do that.  Always use a version number because latest can map to anything.  Every time I do a Docker build and I do a Docker tag, latest effectively can map to that.  So it can change under me.  So if I did want to upgrade my application now, I have one dot zero version running.  I can just change the image name in the deployment config and change the new tag and it will  do a redeployment for me.  All automatically for you.  Storage, I'm running out of time.  So storage, people originally with Docker saw it as being for 12 factor apps or cloud  native apps that didn't have any storage.  If you're using Heroku, you're very, very familiar with that.  If you deploy stuff to Heroku, there is no storage.  If I write to the local file system application, I've lost that.  You always have to have data outside in a separate database or storing stuff in S3 or  other external.  One of the good things about Kubernetes is it supports the concept of persistent volumes.  This is very handy if you need to bring in apps, so-called legacy apps or non-12 factor  apps which do have a requirement for storage or if you need to just support storing of  image uploads in a blog app, for example.  So I can actually mount in persistent volumes into my container.  The way this works is that I send you to tell Kubernetes, I need persistent storage, I need  this amount, and this is the type of storage I need.  This is very important.  So there's different types of storage.  Read write ones, read only many, and read might many.  Now read write ones, if you're familiar with Amazon, that equates to elastic block storage.  Elastic block storage volumes can only be mounted on one node at a time.  Read write many down the bottom, there's traditional files in storage like NFS.  You can mount that wherever you want.  And this is important because if I have an app and I want to scale it up and have multiple  instances, then I can't use the top one because Kubernetes is going to make decisions about  where it runs the instances of my apps and it could run multiple instances across different  nodes.  So if you're using the wrong storage type and you say read write ones, it'll only bring  up one instance or whatever instance it puts on that node.  And this is the really thing about Kubernetes you have to be careful of.  If you can set up a configuration and that configuration could be wrong or be telling  Kubernetes to do something that it can't satisfy.  And when I run that command or update the config, you do not get an error.  Kubernetes will just spin it's wheels because it can't actually satisfy it.  That's one of the weird things.  And in storage, an example, if I use read write one storage, I think it would be mounted  on one node, 10 instances, why are they only at four instances running?  It's because Kubernetes couldn't satisfy that request because it couldn't perhaps put more  instances to wrap on that one node.  It's trying to distribute them elsewhere and it's just sitting there waiting for storage  which you can't get.  So if you do, I start with using EBS on Amazon, you are stuck with one replica or you have  to do tricks like tell Kubernetes to set up all my instances, my app to run on one node.  But then you lose your fault tolerance a bit because if I lose the whole node, I've got  no instances because the idea of Kubernetes will distribute the instances across the nodes.  So if you do lose a node, then it's still got other instances running.  Again, persistent volumes is like Ingress.  You have to start marking around with raw definitions and then you can just create it,  you have to start marking around the deployment config, you can change the deployment config  and say, okay, I need this persistent volume claim for this deployment but I also need  to then say where that is going to be mounted inside of that particular container.  So there's lots of marking around to bring all these things together.  But then once we do have storage, we want to get in there and do things like set up  our super user cell, we can do that.  We can go in there, kubect, run our create super user.  I'm using SQLite and fine, I've got my storage.  If I want to change config, like various systems like this, I can set environment variables  which pass down into configs.  So if I'm using modWiz, I can make that deployment config or redo.  More importantly, environment passing is good for linking up things like databases.  So I could have just passed in all the details of where my database is and environment variables,  but Kubernetes offers another thing called secrets and that is the ability to put a configuration  in what's called a secret and I can actually have that as a separate object and I can use  that from multiple things.  So here I've already pre-deployed my database, it has a secret with my credentials in it  and without needing to know what they are, I can say, okay, set my environment but pull  those variables from the secret and set them all automatically and I can link them together.  In addition to secrets, another thing called config maps, they essentially work exactly  the same but secrets have some better guarantees on how that information is managed such that  it's not stored on disk.  So if I shut down a machine, ship it out, I haven't put my secrets out and given them  away to someone.  And I can get back and view my configuration, you can see where it's coming from.  Now that's the guts of it, that's a very simple example of running an app.  Now I mentioned these distributions earlier, distributions can make things simpler because  they can add on extra bits which make it easier from a developer's perspective.  Kubernetes is very much regarded as being an operations platform.  As developers, that's not great, operations tend to, oh, that's our stuff, we don't want  to touch it.  That's what you might look at some of the distributions, so OpenShift from Red Hat and  OKD which is the upstream version of that and also Pivotal, we're coming over from  CloudFabric, they have mechanisms or tooling which are much more developer friendly and  allow you to build up things.  Now I'm going to skip through a quick example of how to do what I just did but using OpenShift  using their tooling.  It's still Kubernetes underneath.  There's a few little things that are different but I can go into OpenShift and one of the  things here is that in what I did before I had an existing image, here I'm going to  go into OpenShift and say, here is my source code repository on GitHub, deploy it for me  and it's going to do that platform as a service type thing that you've made with Heroku.  It'll take my source code and build into an image for me and it will go off and create  a deployment for me, create a service and get it all running for me.  I can also do things like adding in volume very easily because it has extra commands  in there to very easily claim storage and mount it in straight away.  So I've deployed it, got my service already, it didn't have to expose it and I can add  in my storage and I can then expose it.  So three steps, I've very quickly done all of what I did before.  In this case, some of the things underneath, if I now query it back, you'll see a lot of  the things are a bit different and this is a bit of a historical because when OpenShift  was first created, things like deployment didn't exist in Kubernetes, things like ingress  didn't exist in Kubernetes.  So you might see people talking about, well OpenShift's not really Kubernetes because  it's different.  It's just a history of it.  Things like these didn't exist, so OpenShift went and created their own and that's actually  fed into then Kubernetes in developing things like deployment and ingress in Kubernetes.  OpenShift is in the process of migrating across to using these new ones in there.  But do look at some of the distributions, they do provide simpler tooling for this.  And one further example in OpenShift is that if I want to deploy a database, there is,  in OpenShift has a concept of templates in there and I can say here, create me database  and that's all I need to do.  It'll go and create it, create me credentials, create that secret which I can then bring  it from environment into map, just like that.  There's also have web consoles which are much more developer friendly.  A default plain vanilla communities doesn't really have a console.  There is a management console you can deploy yourself.  All that stuff comes out of the box with OpenShift.  And it's here, I'm running out of time.  So cleaning up, I've mentioned labels before, it's been very important.  And that's where it comes in, important, deleting stuff.  You use your labels for deleting things.  I mentioned templates.  There are other templing systems too to look at, Helm and Ksonnet.  There are others as well.  They help you to essentially define your application as a whole lot of templates and they to vary  easily to reproducible deployments.  So look at those.  Building, you may be familiar with Docker builds because that all becomes standards under the  OpenCotange initiative.  There are now alternative tools for doing builds.  Builder is an example of doing build.  Has a companion program called Scopio for doing pushing images and so on.  And they're all right together with thing called Podman.  The source to image is what OpenShift uses for doing its past type builds.  Very quick one, don't run up, don't design your app images though to run as root.  Please don't do that.  Run them so they can run as arbitrary ideas, user ideas much better.  And there's various tricks.  I'll skip this because I'm running out of time.  You can go back.  If you want to play with it, some resource we can go.  Kubernetes by example is a great place we can go to learn about some of the low level  concepts in Kubernetes, what they are for, how to use them.  There's a site called Catechord where you can go and it will spin up an instance of  Kubernetes for you.  And it will have on one side a set of instructions, a terminal, and you can go through and just  do it all on the spot.  Just click on the instructions and it will run them all for you.  You can play with Kubernetes that way.  If you want to run Kubernetes yourself on your own box, it's Minikube.  So rather than go and try and do the whole hog of setting up a whole cluster, you can  use Minikube.  It just runs up inside of VM.  And there are similar things for OpenShift.  It has Catechord environment as well, which is learnopenshift.com.  It has Minikshift, same as Minikube.  And finally, if you want resources, I've got my name on a couple of books if you want to  look at this from OpenShift side more.  And they are free downloads.  And that's it.  I don't think we've got any time for questions.  So I'll be over.  You can get me later.  No time for questions.  If Graham's willing to take questions in the hall afterwards, you can find them then.  So thank you, Graham.  Thank you.
DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_0_119.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_120_251.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_252_335.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_336_407.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_408_1835.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_1836_2651.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_2652_3455.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_3456_5123.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_5124_6827.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_6828_10883.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_10884_12839.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_12840_13643.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_13644_13919.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_13920_15671.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_15672_17831.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_17832_22631.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_22632_24659.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_24660_25259.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_25260_25523.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_25524_25859.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_25860_27047.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_27048_28031.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_28032_29867.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_29868_31391.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_31392_31943.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_31944_32399.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_32400_34523.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_34524_37295.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_37296_38627.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_38628_39827.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_39828_41831.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_41832_48023.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_48024_49739.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_49740_50411.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_50412_51191.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_51192_51491.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_51492_53387.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_53388_53771.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_53772_55427.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_55428_56267.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_56268_57371.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_57372_61211.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_61212_61895.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_61896_62075.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_62076_62171.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_62172_62567.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_62568_62963.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_62964_63299.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_63300_64643.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_64644_64799.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_64800_66059.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_66060_66851.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_66852_67283.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_67284_67343.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_67344_67595.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_67596_68579.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_68580_69071.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_69072_69155.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_69156_69527.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_69528_69791.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_69792_69971.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_69972_70175.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_70176_70223.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_70224_70775.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_70776_71195.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_71196_71255.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_71256_71543.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_71544_71987.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_71988_72263.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_72264_72407.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_72408_72467.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_72468_73067.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_73068_73223.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_73224_73271.png DjangoCon_US_2018_-_Fundamentals_of_Kubernetes_for_Django_developers_by_Graham_Dumpleton_kbarwyun-c4_73272_73336.png