Cloud Native Python in Kubernetes
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Bruynooghe, Floris
Cloud Native Python in Kubernetes Serverside applications are more and more likely to need to run in dynamic cloud environments where they can automatically scale as required. One rightfully popular approach is to run the application as a Docker container inside a Kubernetes cluster, giving you a lot of operational benefits thanks to the Kubernetes folks. For the most part it is rather easy to make your Python application work inside a Docker container. But there are a number of common patterns one can follow to save time by delegating more things to the runtime environment. Furthermore you can start adding a few simple non-intrusive features to your application which will help improve the application live-cycle in the cluster, ensuring smooth hand-over when migrating the container to different nodes or scaling it up or down. This talk will quickly cover the basics of Kubernetes and will then start from a simple program and will discuss the steps to take to make it behave well in this environment. Starting with the basics steps you can rely on the runtime for, covering logging and all the way to supporting the service life-cycle, health checking and monitoring in a Kubernetes environment. You will see that building a cloud-native application is not very hard and something you can gradually introduce
 Good afternoon. So yeah, I'm Flores. I've been a Python developer for quite a while now. In my spare time, I'm quite often, sorry, can you, can everyone hear me properly? Should I stand closer? Is that better? Okay. Sorry. Okay. So yeah, I've been involved in Python for quite a while. I work on open source as well, kind of usually around PyTest, et cetera. I have very recently changed jobs and I'm now a site reliability engineer at Google. While Kubernetes is a Google, originally came out of Google, this is not actually a Google project, Google Talk, sorry. This is largely based on my experience from my previous job where I was working with microservices in Python that we ran on Kubernetes. A little note about the title. So Kubernetes is part of the cloud native computing foundation, CNCF, and that's kind of where the title kind of came from, so cloud native Python. Okay. So this is kind of the contents that we'll be covering. So I'll start with a very, really, really brief introduction about Kubernetes, probably the shortest introduction you can ever have. But hopefully that should be, if you're not familiar with it yet, that should be enough to kind of follow the rest of the slides, really. And I'll kind of introduce a little example, just a kind of traditional echo server, so it's a network server, you send the message, you get the same message back. And then we'll kind of take that example throughout the rest of the talk to sort of start modifying it a little bit. So introduction to Kubernetes, like this is kind of Kubernetes in one slide, which is a tricky thing to do, I guess. So the idea of Kubernetes is that it's a cluster or orchestrator, really. So the idea is that you give it a bunch of machines, and it creates a cluster out of it, and then when you want to run your application, you just say, run my application somewhere in this cluster. And Kubernetes itself will decide where the right place in that cluster is for your application to run. And the kind of the aim that you're aiming for is to, the reason we want to build an architect, a system like this, especially with multiple microservices, and you have multiple instances of every service that tries to make a really resilient application. So if one of the machines in the cluster is unhealthy or something, you can just take it down, you can fix it, replace it. And your application just keeps running. So if something crashes, request will just be rooted somewhere else, and you create a really resilient and always up kind of application. That's the end goal of what you're trying to do. We're trying to run in Kubernetes. So the core concepts of Kubernetes is like, you want to run your application, and your application, in our case, is just going to be a Python application. And Kubernetes runs kind of containers, essentially. So you need to containerize your application. So right now, that basically means Docker. In the future, hopefully, that will be also like a rocket and the like. So you create a container out of your application. I'll skip over that part today. And Kubernetes runs this inside of a pod. So pod is kind of the smallest unit that Kubernetes will run for you. Essentially it's just another app for your container. You don't really have to worry why they decided to create a pod. It's kind of the idea to treat multiple containers inside one pod and treat it as one unit. Why you want to do that doesn't really matter for today. So pod, you then tell Kubernetes, please run this pod for me, which is your application. Kubernetes will kind of just find a server in its cluster and start to run it for you. But the problem with pods is that they're kind of ephemeral. So if the pod gets killed for some reason, an administrator goes rogue or a machine gets taken down or something like that, then your pod is gone. So that's a far cry from our resilient application. So the next concept that Kubernetes introduces is kind of this idea of a replica set. And a replica set is essentially we'll kind of continuously look at what's running inside of Kubernetes. And the idea is that in your replica set, you say, I want to run this many instances of my application. And whenever the replica set sees that your application isn't running that many times, it will try and make sure that that happens. So if there's not enough instances of your application, then it will create more runs. If there's too many, it will kill a few. So that means that machine gets taken down, the replica set will notice this, and will create a new pod instead. So this starts to create your application to be always there, especially if you request multiple instances. The problem then is, of course, again, these pods, they just run on a random machine somewhere in your cluster. You don't know anything about them. You don't know how to contact them. So that's where the concept of the service comes in. So the service is essentially some sort of a fixed IP address. It's the easiest way to think of inside your cluster. And if you need to contact your application, you can contact it via the service. And the service will essentially load balance between all the instances of your pod, and it will make sure if you send traffic to the service, it will go to one of the instances of your application that's running somewhere in the cluster. And these are really the basic things about Kubernetes that's kind of enough to follow. Anyway, I'm now going to say everything else is bells and whistles. There is lots of other layers. They don't actually even recommend you use replicats at the moment. So there's obviously lots more going on. But that's the core concept, and that's a core concept that allows you to create resilient applications. So that should be enough for today, hopefully. So this is my little example application that I'll work with to address. It's basically an echo server. So on the network, I've implemented it using ZerumQ because any time you think of creating a TCP socket and you want to send and receive data, you should really use ZerumQ instead because it takes care of a lot of the integrity, networking details, and you get whole messages automatically delivered to your application. You don't have to worry about everything else, really. Other than that, this is fairly standard. I should point out that all the code I show is kind of slideware. So I use globals. I don't show all the imports. You shouldn't write application like this, obviously. So yeah, this application, basically, the main loop, essentially, is like, I create my socket that I want to listen on. I then use this polar thing, which is basically if you've done normal tspe programming, select. So essentially, it's asking the operating system, like, can I sleep until the next message is available and then when the next message is available, that will kind of come as like an event. And then I basically pass the server socket, the echo server socket to the event handler for that. The event handler will then basically receive the message and send it back to me. So that's all this is. This is basically an infinite loop receiving and sending the messages. So here are the few little helper functions to make it fit on the slide. So create and bind, really nothing, just plumbing. So create a socket, bind it so that we actually can receive connections. And we register it also into the polar, again, using globals. So try this. It's not nice code. And the handler is where the actual work happens. So we receive the message. And do a little bit of Xerium Q bookkeeping to split off the way Xerium Q passes you the peer address. And then log the message and then send it back to whoever sent it to us. So that's kind of the very simple application. And the first thing to kind of notice is that that's actually kind of sufficient. So we can just take that application, put like, you know, if name equals main thing in there, et cetera. And we can just containerize that and run that in our Kubernetes cluster. And that will just work. So the first thing to sort of rely on when you're in Kubernetes is sort of just relying on the fact that you are running in an environment. So you don't have to put complexity in your application. And it allows you to really have a very simple logical flow, which is kind of what we just saw was like a super simple straightforward internal architecture. And that's kind of true for larger applications. Well, you can sort of skip through all the boilerplate, et cetera. And the first thing in that is like, I didn't write any exception handles in that code. And so it fits on a slide. But that's actually generally quite true. You don't really have to worry about exceptions because your application runs in multiple instances. And if you do get something unexpected, like, I don't know, maybe the bind that could fail to the socket because whatever goes wrong on the machine, like, I don't really mind. My process will die and Kubernetes will just make sure that a new one gets created somewhere else instead. And it will probably work. So yeah, so you don't have to worry about that especially. You can even go as far as kind of doing that for when you're receiving a request from someone or the service. If this service is completely internal to yourself, so you would actually, or your team or whatever is the author of the service, then essentially you can treat that failed request as a bug. Sorry, invalid kind of schema of the request kind of as a bug. And you can basically crash again. If you're doing this for external application, so if you're actually receiving user requests, then that is probably a bit too eager, like, too brittle. So in that case, you probably do want to catch the exception for request validation. Because there is a little, there is, depending on application, I mean, in this case, not very much, but there is some overhead in starting up your application again. And the other thing that kind of happens when you just crash and that you should take into account is that if you have network connection, so you're receiving messages from some clients, those network connections, they will have buffers in them. So basically there will be requests queued up already in your local process. So your queue makes that really obvious because there is explicitly a queue with your socket. But even if you use raw TCP, there will always be internal kernel buffers. The kernel may even have accepted new requests already and just queue them up, even though your application has no ID yet. There might be data still on the wire just coming over. And if you just crash, then you kind of lose that data. So those requests and whoever created those requests will have to then kind of wait and time out and retry. So that's not very good. Yeah, so you want to take that kind of into account. And that kind of brings you to how do I organize my messaging, et cetera, and you can kind of start playing if you really don't want to suffer from that. You can start playing with message brokers like RabbitMQ or different systems like Kafka, et cetera. But these all come with trade-offs expenses. The only thing I would say in here is basically be aware of when you crash, you may lose requests and make sure that that's OK in the system you're designing. So in this slide, this actually shows how you actually would create your, this is the recommended way that they want you to create your pods. Deployment is essentially just a wrapper around this replica setting. The reason they have it is because it creates updating your application slightly easier. But as far as from our point of view, the two important things here is this line that says replica three. That means we are requesting three replicas. So Kubernetes will always ensure that there's three of us running. And the other really important thing is the very last line that says restart policy always. That line will basically, that tells Kubernetes that if we're crashing, just start us again, please. So that's kind of the first step. The second thing is the script had no concurrency. And while it was a very simple example, this is generally true. You can rely on, you can keep your internal code really simple because the idea is to scale via the process model. It's some sort of, if you've ever heard of 12 factor apps kind of methodology. That's kind of the idea of you just create more, you scale horizontally by creating more instances of our application. We just saw that's kind of what you already did. We just create more replicas and they will handle the traffic. That means that internally we can have really easy debugging, et cetera, because our control flow just gets really simple and we don't worry about any of the other stuff. Yeah, and basically your server is your load balancer in this case. So this is kind of the service definition that we have. One thing again, one gotcha to look out for with this sort of load, the service that you create, which load balances your traffic between your pods is if your protocol that you're using uses long standing connections, which again, like Xerium Q in our example points is out very well because Xerium Q creates long standing connections. You connect and then it tries to reuse that connection for lots of requests and responses. So our echo service will accept lots of echo requests from the client on one connection. So this means that essentially we don't get the load balancing. So one client will permanently be connected to one of the pods or one of our applications essentially. And so that's not very load balancing. Well on the other hand, if you're using HTTP, then you get a new TCP connection gets created for each request and that would distribute automatically. The trick to use there is that Kubernetes actually allows you to see the layer below services and it has end points, objects, it calls end points there. And that actually allows you to see which end points, basically IP addresses that are part of your service. And using that information so we could update our application to sort of query the Kubernetes API to ask what end points do I actually want to connect to. You can tell Xerium Q connect to all these end points. The downside is that you have to kind of work with the Kubernetes API. You need to be constantly aware that this can change. So when an end point disappears, you need to tell Xerium Q connect from the end point, et cetera. But that's just generally the sort of thing you need to be aware of when you have long protocols that use long standing connections, basically. Next on, talking a bit about logging. So you may have seen in the very first example, I encouraged that this print statement. So by default, Docker kind of takes standard output of your container as log data. And Kubernetes will again take the log data from the Docker containers and they'll make that available to use. And generally the idea is that at operations time, you sort of hook that up to a log aggregation of some sort, something like elastic search or flu indeed or something like that. But using simple print statements is not very nice. In general, you want to be able to control log levels at command line level. Again, that's sort of a 12-hectare app kind of thing. So you really want to use logging libraries. Yeah. So logging libraries are sort of common, there's quite a few variations. And they all kind of try to wrap a horrible amount of global state into a nice API. And global state is always quite horrible. So they all kind of are ugly in one way or another. I quite like logbook. That's not kind of like the way it tries to handle the global state. But there's nothing in there that's really about logbook than standard library logging, if you like standard library logging. One nice thing about logbook is that you can use the normal curly braces formatting to a new style formatting instead of percent formatting from a standard library logging. But the main thing here is to notice that once you start using a logging library, you can actually start, you can hook up your logging library to instead of printing out to standard out, which is kind of the first thing you probably do because if you don't have all the infrastructure yet. But you can hook it up to send the log records directly to the aggregator. And this allows you to, like if you have tracebacks or something, you can send them as a single big block. So the next thing that you really want to do as soon as you start using a logging library is wrap your main application into basically this exception logging. Our logging libraries will support some sort of variation of this. But the idea here is that by doing this, and make sure that any unhandled exception, as I was advertising earlier, will be captured by the logging library and will be sent as one single log record back to your logging aggregator. So next on is kind of this concept that Kubernetes called health endpoints. And the idea here is that, the central idea here is that when your application starts up, Kubernetes has said, you know, start this container. And as soon as that container is kind of running from a process point of view, it's available. So the service object that you created for your application will start sending traffic to you. But there is a finite amount of time that your application is running and you haven't opened your socket yet and you're not listening for connections yet. In this case, obviously, it will be very small. But if you have a lot of setup and things you have to do before you start accepting connections from clients, that delay might be bigger as well. And the problem is, if at that point, Kubernetes is sending traffic to your application, then you're basically saying, you've used connection. And all these clients are like, well, the service is down. So you don't want to do that. And what Kubernetes does is it introduces these readiness probes. And the idea is basically that, Kubernetes, like, after starting your application, it will wait until the probe succeeds. And once the probe succeeds, only then it will start sending traffic to your application. So this is kind of how you configure that in Kubernetes, the AMO configuration. In our case, so there are several different probes you can use. In our case, it's just a simple echo server. So all we care about, really, from the probes point of view is that the socket is open and listening. So this TCP socket probe essentially says, as soon as you have, as soon as the socket accepts connections, send me traffic. Kubernetes will literally just try and connect to the socket. And as soon as there's a connection close, and say, yeah, OK, I'll send the traffic. So that means there's still kind of a tiny delay in which we might actually have bound our socket, but not yet in the main loop of pressing some messages. But because we know the buffering and the queuing in our sockets, to care of this, that's basically perfectly fine. That will be a short amount of time. We'll have some requests get queued up, but we'll start serving them very shortly. If your service is actually using HTTPS transported, it kind of has built-in support for that. And there is this convention of using this health Z root. And basically, this is really nice in a way, because you can basically tell your, the readiness probe is kind of completely in line with, or can be, if you create a search, to be completely in line with your normal request processing. So if that actually returns, so basically, Kubernetes only looks for the 200 OK on there. So as soon as that returns 200 OK, it will start sending you traffic. And because you can do this completely in line, you actually have quite high assurance that, yes, my application is fully running and can start serving traffic. So the same concept of kind of readiness. So the same concept also happens for, during the pod's lifetime, really. So when your application is running, especially when you're running your application at a large scale and it has lots of requests, sometimes things will just go wrong. And one of the many instances will start misbehaving. This might be because of external things. Like there's another container on that box that, even though things should be isolated, they're not always as isolated as you would imagine. And things go terribly wrong, or someone decided to run a backup on that machine, and things go really slow, or all sorts of things happen, right? You're really... And the problem... So the idea of readiness probes is that you don't want to be sending requests to an application that's slow to respond, or it's just completely stuck or something like that. It does this in essentially a very similar way. So it just has this liveness probes thing. And this shows kind of the third type of probe that you can use. So we've already seen like TCP socket and HTTP GET. And here I decided to use the exec one. Exec is kind of the most work, but it's also the most flexible. So the problem is like the TCP one that we used for readiness, and it was very suitable there, it's not very useful for the liveness probe. And what we really want is kind of the same as you can use with HTTP GET. We want to know in line, like, are things working correctly. So the trick you do there is this exec command basically says, it gives you the command line that you want to execute inside of your container. So you have to provide an extra binary inside of your container, which will ideally tell how healthy or not healthy that pod is. And ideally, you want to aim for this in line checking. So that's exactly what I've done here. I've added an extra script in my application, and I just invoke it with Python. So this is kind of how simple the script can be. Again, obviously, I have a very simple application. But literally, it just creates a socket connected to the public endpoint. In this case, because I'm running on the same container, it's actually on local host, but we still make the TCP connection. So it still gives us a very high kind of idea of what the public behavior is of our application. And we just send a message, and we wait for, like, 500 milliseconds. And if you haven't got a message back yet at that time, then we fail. And then Kubernetes will start taking our pod out of, basically, stop sending traffic to our pod and send it to the healthy ones instead. You may notice I'm not actually even receiving my response. I just check, hey, there is a response. In this case, I think that is sufficient. So it depends on the protocol you have or what sort of messaging you use. Sometimes you might want to introduce slightly more. And often it is nice to build in something like that, what the hell said kind of root to that in the H2B convention. Just something that, you know, it is working. Yes, just something that, you know, it is working. And everything is fine. So that's the idea of lightness. So next on is, like, termination. And again, this is, like, very similar. If you thought about how we, you know, at startup time, we didn't want to refuse any connections. At termination time, we don't want to drop any connections that exist. So any requests that are currently queued up on our process, if we just say, oh, we have been asked, if we don't do anything, basically, our application will just receive termination signal and it will just die. And all the requests that are queued up will basically be lost and all the clients will be waiting again. So that's not very good. So instead, we should handle the termination signal, which is done in communities just like it's always been done in Unix systems. You basically get, yeah, via Docker, but you get sick terms sent to you. And the signal handler here that we create is just a very simple internal socket. It's essentially, I think, of it as a pipe. And all we want to do is, like, send this signal, like, I'm sending a single byte to my main loop and then my main loop knows it has to shut down. And yeah, and it can shut down while trying to handle the connections. I'm using the same signal handler here for both sick term and sick int. Sick int is the signal that you get when you press control C normally when you're running on a terminal. So in Python, that normally gets automatically translated to keyboard interrupt exception because you want the application to behave in the same way when you're running on the test. It's usually best practice to just bind both the signal handlers. The modifications for my main loop are maybe a little bit more messy now. But essentially, it's not that much. I just, again, create, instead of creating just one socket in my main loop, I create two sockets. So I add this termination socket. I add it to the potler. And when I now receive, the important thing here is when I receive this new, this single byte, basically, I don't even care what that byte is, by the way. And I just know, oh, I received a message from my termination socket, so I need to shut down. And here, I first unbind. So this means that I will stop receiving new connections. And then I basically keep processing. While there are still events in the queue, I keep processing. And I have also, I set a time out actually to like five seconds, something which may be fairly high. But the idea here is that some requests might actually already be on the wire. And I want to give those a chance to be processed well. Once there are no longer any messages in my queue, the while loop will finish every time. So the last thing that I would like to add is monitoring. So Prometh is another cloud-native computing foundation project, actually, which is kind of why I mentioned it. And the idea here is to start, you always want to know kind of what's going on. And Prometh is this kind of, obviously, this option of doing white box monitoring in a way. So you can add counters, and you can add metrics inside your application. You can start enriching your metrics collection, basically. So Prometh just works in a very, in a pool-based fashion. So you have the central infrastructure, and it will go around to all your services. And it will do an HTTP request and get back your metrics, basically. It's a little bit like SNMP over HTTP, if you remember the SNMP pool medal. But at least we learned that monitoring data is probably even more important than production traffic. So we actually use a reliable TCP transport instead of something like UDP. Anyway, Prometh is obviously a very difficult and big project, but the idea here that I really want to show is just how easy it is to get started with it. And you can just add very little things. There's a last year that was a talk about, from Hinnick, I think, at Europe, at Python, which goes in a lot more detail about Prometh. So this is kind of, yeah, the recap. The idea is basically, none of this is actually strictly required to be able to run on Kubernetes. You can adopt this gradually as you need. And yeah, just keep your architecture simple. Think about when you lose requests. And you don't want to go blind, basically. So you always want to have some instrumentation and monitoring. Thank you very much. I think I'm about out of time. So unfortunately, there probably won't be any questions, but you can find me outside if you like. Thank you.
 Good afternoon.  So yeah, I'm Flores.  I've been a Python developer for quite a while now.  In my spare time, I'm quite often, sorry, can you, can everyone hear me properly?  Should I stand closer?  Is that better?  Okay.  Sorry.  Okay.  So yeah, I've been involved in Python for quite a while.  I work on open source as well, kind of usually around PyTest, et cetera.  I have very recently changed jobs and I'm now a site reliability engineer at Google.  While Kubernetes is a Google, originally came out of Google, this is not actually a Google  project, Google Talk, sorry.  This is largely based on my experience from my previous job where I was working with microservices  in Python that we ran on Kubernetes.  A little note about the title.  So Kubernetes is part of the cloud native computing foundation, CNCF, and that's kind  of where the title kind of came from, so cloud native Python.  Okay.  So this is kind of the contents that we'll be covering.  So I'll start with a very, really, really brief introduction about Kubernetes, probably  the shortest introduction you can ever have.  But hopefully that should be, if you're not familiar with it yet, that should be enough  to kind of follow the rest of the slides, really.  And I'll kind of introduce a little example, just a kind of traditional echo server, so  it's a network server, you send the message, you get the same message back.  And then we'll kind of take that example throughout the rest of the talk to sort of start modifying  it a little bit.  So introduction to Kubernetes, like this is kind of Kubernetes in one slide, which is  a tricky thing to do, I guess.  So the idea of Kubernetes is that it's a cluster or orchestrator, really.  So the idea is that you give it a bunch of machines, and it creates a cluster out of  it, and then when you want to run your application, you just say, run my application somewhere  in this cluster.  And Kubernetes itself will decide where the right place in that cluster is for your application  to run.  And the kind of the aim that you're aiming for is to, the reason we want to build an  architect, a system like this, especially with multiple microservices, and you have multiple  instances of every service that tries to make a really resilient application.  So if one of the machines in the cluster is unhealthy or something, you can just take  it down, you can fix it, replace it.  And your application just keeps running.  So if something crashes, request will just be rooted somewhere else, and you create a  really resilient and always up kind of application.  That's the end goal of what you're trying to do.  We're trying to run in Kubernetes.  So the core concepts of Kubernetes is like, you want to run your application, and your  application, in our case, is just going to be a Python application.  And Kubernetes runs kind of containers, essentially.  So you need to containerize your application.  So right now, that basically means Docker.  In the future, hopefully, that will be also like a rocket and the like.  So you create a container out of your application.  I'll skip over that part today.  And Kubernetes runs this inside of a pod.  So pod is kind of the smallest unit that Kubernetes will run for you.  Essentially it's just another app for your container.  You don't really have to worry why they decided to create a pod.  It's kind of the idea to treat multiple containers inside one pod and treat it as one unit.  Why you want to do that doesn't really matter for today.  So pod, you then tell Kubernetes, please run this pod for me, which is your application.  Kubernetes will kind of just find a server in its cluster and start to run it for you.  But the problem with pods is that they're kind of ephemeral.  So if the pod gets killed for some reason, an administrator goes rogue or a machine gets  taken down or something like that, then your pod is gone.  So that's a far cry from our resilient application.  So the next concept that Kubernetes introduces is kind of this idea of a replica set.  And a replica set is essentially we'll kind of continuously look at what's running inside  of Kubernetes.  And the idea is that in your replica set, you say, I want to run this many instances  of my application.  And whenever the replica set sees that your application isn't running that many times,  it will try and make sure that that happens.  So if there's not enough instances of your application, then it will create more runs.  If there's too many, it will kill a few.  So that means that machine gets taken down, the replica set will notice this, and will  create a new pod instead.  So this starts to create your application to be always there, especially if you request  multiple instances.  The problem then is, of course, again, these pods, they just run on a random machine somewhere  in your cluster.  You don't know anything about them.  You don't know how to contact them.  So that's where the concept of the service comes in.  So the service is essentially some sort of a fixed IP address.  It's the easiest way to think of inside your cluster.  And if you need to contact your application, you can contact it via the service.  And the service will essentially load balance between all the instances of your pod, and  it will make sure if you send traffic to the service, it will go to one of the instances  of your application that's running somewhere in the cluster.  And these are really the basic things about Kubernetes that's kind of enough to follow.  Anyway, I'm now going to say everything else is bells and whistles.  There is lots of other layers.  They don't actually even recommend you use replicats at the moment.  So there's obviously lots more going on.  But that's the core concept, and that's a core concept that allows you to create resilient  applications.  So that should be enough for today, hopefully.  So this is my little example application that I'll work with to address.  It's basically an echo server.  So on the network, I've implemented it using ZerumQ because any time you think of creating  a TCP socket and you want to send and receive data, you should really use ZerumQ instead  because it takes care of a lot of the integrity, networking details, and you get whole messages  automatically delivered to your application.  You don't have to worry about everything else, really.  Other than that, this is fairly standard.  I should point out that all the code I show is kind of slideware.  So I use globals.  I don't show all the imports.  You shouldn't write application like this, obviously.  So yeah, this application, basically, the main loop, essentially, is like, I create  my socket that I want to listen on.  I then use this polar thing, which is basically if you've done normal tspe programming, select.  So essentially, it's asking the operating system, like, can I sleep until the next message  is available and then when the next message is available, that will kind of come as like  an event.  And then I basically pass the server socket, the echo server socket to the event handler  for that.  The event handler will then basically receive the message and send it back to me.  So that's all this is.  This is basically an infinite loop receiving and sending the messages.  So here are the few little helper functions to make it fit on the slide.  So create and bind, really nothing, just plumbing.  So create a socket, bind it so that we actually can receive connections.  And we register it also into the polar, again, using globals.  So try this.  It's not nice code.  And the handler is where the actual work happens.  So we receive the message.  And do a little bit of Xerium Q bookkeeping to split off the way Xerium Q passes you the  peer address.  And then log the message and then send it back to whoever sent it to us.  So that's kind of the very simple application.  And the first thing to kind of notice is that that's actually kind of sufficient.  So we can just take that application, put like, you know, if name equals main thing  in there, et cetera.  And we can just containerize that and run that in our Kubernetes cluster.  And that will just work.  So the first thing to sort of rely on when you're in Kubernetes is sort of just relying  on the fact that you are running in an environment.  So you don't have to put complexity in your application.  And it allows you to really have a very simple logical flow, which is kind of what we just  saw was like a super simple straightforward internal architecture.  And that's kind of true for larger applications.  Well, you can sort of skip through all the boilerplate, et cetera.  And the first thing in that is like, I didn't write any exception handles in that code.  And so it fits on a slide.  But that's actually generally quite true.  You don't really have to worry about exceptions because your application runs in multiple  instances.  And if you do get something unexpected, like, I don't know, maybe the bind that could fail  to the socket because whatever goes wrong on the machine, like, I don't really mind.  My process will die and Kubernetes will just make sure that a new one gets created somewhere  else instead.  And it will probably work.  So yeah, so you don't have to worry about that especially.  You can even go as far as kind of doing that for when you're receiving a request from someone  or the service.  If this service is completely internal to yourself, so you would actually, or your team  or whatever is the author of the service, then essentially you can treat that failed  request as a bug.  Sorry, invalid kind of schema of the request kind of as a bug.  And you can basically crash again.  If you're doing this for external application, so if you're actually receiving user requests,  then that is probably a bit too eager, like, too brittle.  So in that case, you probably do want to catch the exception for request validation.  Because there is a little, there is, depending on application, I mean, in this case, not  very much, but there is some overhead in starting up your application again.  And the other thing that kind of happens when you just crash and that you should take into  account is that if you have network connection, so you're receiving messages from some clients,  those network connections, they will have buffers in them.  So basically there will be requests queued up already in your local process.  So your queue makes that really obvious because there is explicitly a queue with your socket.  But even if you use raw TCP, there will always be internal kernel buffers.  The kernel may even have accepted new requests already and just queue them up, even though  your application has no ID yet.  There might be data still on the wire just coming over.  And if you just crash, then you kind of lose that data.  So those requests and whoever created those requests will have to then kind of wait and  time out and retry.  So that's not very good.  Yeah, so you want to take that kind of into account.  And that kind of brings you to how do I organize my messaging, et cetera, and you can kind  of start playing if you really don't want to suffer from that.  You can start playing with message brokers like RabbitMQ or different systems like Kafka,  et cetera.  But these all come with trade-offs expenses.  The only thing I would say in here is basically be aware of when you crash, you may lose  requests and make sure that that's OK in the system you're designing.  So in this slide, this actually shows how you actually would create your, this is the  recommended way that they want you to create your pods.  Deployment is essentially just a wrapper around this replica setting.  The reason they have it is because it creates updating your application slightly easier.  But as far as from our point of view, the two important things here is this line that  says replica three.  That means we are requesting three replicas.  So Kubernetes will always ensure that there's three of us running.  And the other really important thing is the very last line that says restart policy always.  That line will basically, that tells Kubernetes that if we're crashing, just start us again,  please.  So that's kind of the first step.  The second thing is the script had no concurrency.  And while it was a very simple example, this is generally true.  You can rely on, you can keep your internal code really simple because the idea is to  scale via the process model.  It's some sort of, if you've ever heard of 12 factor apps kind of methodology.  That's kind of the idea of you just create more, you scale horizontally by creating more  instances of our application.  We just saw that's kind of what you already did.  We just create more replicas and they will handle the traffic.  That means that internally we can have really easy debugging, et cetera, because our control  flow just gets really simple and we don't worry about any of the other stuff.  Yeah, and basically your server is your load balancer in this case.  So this is kind of the service definition that we have.  One thing again, one gotcha to look out for with this sort of load, the service that you  create, which load balances your traffic between your pods is if your protocol that you're  using uses long standing connections, which again, like Xerium Q in our example points  is out very well because Xerium Q creates long standing connections.  You connect and then it tries to reuse that connection for lots of requests and responses.  So our echo service will accept lots of echo requests from the client on one connection.  So this means that essentially we don't get the load balancing.  So one client will permanently be connected to one of the pods or one of our applications  essentially.  And so that's not very load balancing.  Well on the other hand, if you're using HTTP, then you get a new TCP connection gets created  for each request and that would distribute automatically.  The trick to use there is that Kubernetes actually allows you to see the layer below  services and it has end points, objects, it calls end points there.  And that actually allows you to see which end points, basically IP addresses that are  part of your service.  And using that information so we could update our application to sort of query the Kubernetes  API to ask what end points do I actually want to connect to.  You can tell Xerium Q connect to all these end points.  The downside is that you have to kind of work with the Kubernetes API.  You need to be constantly aware that this can change.  So when an end point disappears, you need to tell Xerium Q connect from the end point,  et cetera.  But that's just generally the sort of thing you need to be aware of when you have long  protocols that use long standing connections, basically.  Next on, talking a bit about logging.  So you may have seen in the very first example, I encouraged that this print statement.  So by default, Docker kind of takes standard output of your container as log data.  And Kubernetes will again take the log data from the Docker containers and they'll make  that available to use.  And generally the idea is that at operations time, you sort of hook that up to a log aggregation  of some sort, something like elastic search or flu indeed or something like that.  But using simple print statements is not very nice.  In general, you want to be able to control log levels at command line level.  Again, that's sort of a 12-hectare app kind of thing.  So you really want to use logging libraries.  Yeah.  So logging libraries are sort of common, there's quite a few variations.  And they all kind of try to wrap a horrible amount of global state into a nice API.  And global state is always quite horrible.  So they all kind of are ugly in one way or another.  I quite like logbook.  That's not kind of like the way it tries to handle the global state.  But there's nothing in there that's really about logbook than standard library logging,  if you like standard library logging.  One nice thing about logbook is that you can use the normal curly braces formatting to  a new style formatting instead of percent formatting from a standard library logging.  But the main thing here is to notice that once you start using a logging library, you  can actually start, you can hook up your logging library to instead of printing out to standard  out, which is kind of the first thing you probably do because if you don't have all  the infrastructure yet.  But you can hook it up to send the log records directly to the aggregator.  And this allows you to, like if you have tracebacks or something, you can send them as a single  big block.  So the next thing that you really want to do as soon as you start using a logging library  is wrap your main application into basically this exception logging.  Our logging libraries will support some sort of variation of this.  But the idea here is that by doing this, and make sure that any unhandled exception, as  I was advertising earlier, will be captured by the logging library and will be sent as  one single log record back to your logging aggregator.  So next on is kind of this concept that Kubernetes called health endpoints.  And the idea here is that, the central idea here is that when your application starts  up, Kubernetes has said, you know, start this container.  And as soon as that container is kind of running from a process point of view, it's available.  So the service object that you created for your application will start sending traffic  to you.  But there is a finite amount of time that your application is running and you haven't  opened your socket yet and you're not listening for connections yet.  In this case, obviously, it will be very small.  But if you have a lot of setup and things you have to do before you start accepting connections  from clients, that delay might be bigger as well.  And the problem is, if at that point, Kubernetes is sending traffic to your application, then  you're basically saying, you've used connection.  And all these clients are like, well, the service is down.  So you don't want to do that.  And what Kubernetes does is it introduces these readiness probes.  And the idea is basically that, Kubernetes, like, after starting your application, it  will wait until the probe succeeds.  And once the probe succeeds, only then it will start sending traffic to your application.  So this is kind of how you configure that in Kubernetes, the AMO configuration.  In our case, so there are several different probes you can use.  In our case, it's just a simple echo server.  So all we care about, really, from the probes point of view is that the socket is open and  listening.  So this TCP socket probe essentially says, as soon as you have, as soon as the socket  accepts connections, send me traffic.  Kubernetes will literally just try and connect to the socket.  And as soon as there's a connection close, and say, yeah, OK, I'll send the traffic.  So that means there's still kind of a tiny delay in which we might actually have bound  our socket, but not yet in the main loop of pressing some messages.  But because we know the buffering and the queuing in our sockets, to care of this, that's  basically perfectly fine.  That will be a short amount of time.  We'll have some requests get queued up, but we'll start serving them very shortly.  If your service is actually using HTTPS transported, it kind of has built-in support for that.  And there is this convention of using this health Z root.  And basically, this is really nice in a way, because you can basically tell your, the readiness  probe is kind of completely in line with, or can be, if you create a search, to be completely  in line with your normal request processing.  So if that actually returns, so basically, Kubernetes only looks for the 200 OK on there.  So as soon as that returns 200 OK, it will start sending you traffic.  And because you can do this completely in line, you actually have quite high assurance  that, yes, my application is fully running and can start serving traffic.  So the same concept of kind of readiness.  So the same concept also happens for, during the pod's lifetime, really.  So when your application is running, especially when you're running your application at a  large scale and it has lots of requests, sometimes things will just go wrong.  And one of the many instances will start misbehaving.  This might be because of external things.  Like there's another container on that box that, even though things should be isolated,  they're not always as isolated as you would imagine.  And things go terribly wrong, or someone decided to run a backup on that machine, and things  go really slow, or all sorts of things happen, right?  You're really...  And the problem...  So the idea of readiness probes is that you don't want to be sending requests to an application  that's slow to respond, or it's just completely stuck or something like that.  It does this in essentially a very similar way.  So it just has this liveness probes thing.  And this shows kind of the third type of probe that you can use.  So we've already seen like TCP socket and HTTP GET.  And here I decided to use the exec one.  Exec is kind of the most work, but it's also the most flexible.  So the problem is like the TCP one that we used for readiness, and it was very suitable  there, it's not very useful for the liveness probe.  And what we really want is kind of the same as you can use with HTTP GET.  We want to know in line, like, are things working correctly.  So the trick you do there is this exec command basically says, it gives you the command line  that you want to execute inside of your container.  So you have to provide an extra binary inside of your container, which will ideally tell  how healthy or not healthy that pod is.  And ideally, you want to aim for this in line checking.  So that's exactly what I've done here.  I've added an extra script in my application, and I just invoke it with Python.  So this is kind of how simple the script can be.  Again, obviously, I have a very simple application.  But literally, it just creates a socket connected to the public endpoint.  In this case, because I'm running on the same container, it's actually on local host, but  we still make the TCP connection.  So it still gives us a very high kind of idea of what the public behavior is of our application.  And we just send a message, and we wait for, like, 500 milliseconds.  And if you haven't got a message back yet at that time, then we fail.  And then Kubernetes will start taking our pod out of, basically, stop sending traffic  to our pod and send it to the healthy ones instead.  You may notice I'm not actually even receiving my response.  I just check, hey, there is a response.  In this case, I think that is sufficient.  So it depends on the protocol you have or what sort of messaging you use.  Sometimes you might want to introduce slightly more.  And often it is nice to build in something like that, what the hell said kind of root  to that in the H2B convention.  Just something that, you know, it is working.  Yes, just something that, you know, it is working.  And everything is fine.  So that's the idea of lightness.  So next on is, like, termination.  And again, this is, like, very similar.  If you thought about how we, you know, at startup time, we didn't want to refuse any  connections.  At termination time, we don't want to drop any connections that exist.  So any requests that are currently queued up on our process, if we just say, oh, we  have been asked, if we don't do anything, basically, our application will just receive  termination signal and it will just die.  And all the requests that are queued up will basically be lost and all the clients will  be waiting again.  So that's not very good.  So instead, we should handle the termination signal, which is done in communities just  like it's always been done in Unix systems.  You basically get, yeah, via Docker, but you get sick terms sent to you.  And the signal handler here that we create is just a very simple internal socket.  It's essentially, I think, of it as a pipe.  And all we want to do is, like, send this signal, like, I'm sending a single byte to  my main loop and then my main loop knows it has to shut down.  And yeah, and it can shut down while trying to handle the connections.  I'm using the same signal handler here for both sick term and sick int.  Sick int is the signal that you get when you press control C normally when you're running  on a terminal.  So in Python, that normally gets automatically translated to keyboard interrupt exception  because you want the application to behave in the same way when you're running on the  test.  It's usually best practice to just bind both the signal handlers.  The modifications for my main loop are maybe a little bit more messy now.  But essentially, it's not that much.  I just, again, create, instead of creating just one socket in my main loop, I create  two sockets.  So I add this termination socket.  I add it to the potler.  And when I now receive, the important thing here is when I receive this new, this single  byte, basically, I don't even care what that byte is, by the way.  And I just know, oh, I received a message from my termination socket, so I need to shut  down.  And here, I first unbind.  So this means that I will stop receiving new connections.  And then I basically keep processing.  While there are still events in the queue, I keep processing.  And I have also, I set a time out actually to like five seconds, something which may  be fairly high.  But the idea here is that some requests might actually already be on the wire.  And I want to give those a chance to be processed well.  Once there are no longer any messages in my queue, the while loop will finish every time.  So the last thing that I would like to add is monitoring.  So Prometh is another cloud-native computing foundation project, actually, which is kind  of why I mentioned it.  And the idea here is to start, you always want to know kind of what's going on.  And Prometh is this kind of, obviously, this option of doing white box monitoring in a  way.  So you can add counters, and you can add metrics inside your application.  You can start enriching your metrics collection, basically.  So Prometh just works in a very, in a pool-based fashion.  So you have the central infrastructure, and it will go around to all your services.  And it will do an HTTP request and get back your metrics, basically.  It's a little bit like SNMP over HTTP, if you remember the SNMP pool medal.  But at least we learned that monitoring data is probably even more important than production  traffic.  So we actually use a reliable TCP transport instead of something like UDP.  Anyway, Prometh is obviously a very difficult and big project, but the idea here that I  really want to show is just how easy it is to get started with it.  And you can just add very little things.  There's a last year that was a talk about, from Hinnick, I think, at Europe, at Python,  which goes in a lot more detail about Prometh.  So this is kind of, yeah, the recap.  The idea is basically, none of this is actually strictly required to be able to run on Kubernetes.  You can adopt this gradually as you need.  And yeah, just keep your architecture simple.  Think about when you lose requests.  And you don't want to go blind, basically.  So you always want to have some instrumentation and monitoring.  Thank you very much.  I think I'm about out of time.  So unfortunately, there probably won't be any questions, but you can find me outside  if you like.  Thank you.
Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_0_59.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_60_107.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_108_155.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_156_203.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_204_287.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_288_515.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_516_599.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_600_707.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_708_755.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_756_1295.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_1296_1439.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_1440_2039.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_2040_2471.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_2472_2687.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_2688_2939.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_2940_2987.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_2988_3035.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_3036_3239.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_3240_3299.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_3300_9971.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_9972_12227.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_12228_13343.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_13344_14435.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_14436_17783.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_17784_17903.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_17904_17975.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_17976_20195.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_20196_24431.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_24432_25895.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_25896_28391.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_28392_30299.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_30300_33011.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_33012_34571.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_34572_36371.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_36372_38519.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_38520_39407.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_39408_40967.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_40968_42539.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_42540_44855.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_44856_45443.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_45444_45683.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_45684_45731.png Floris_Bruynooghe_-_Cloud_Native_Python_in_Kubernetes_ZKPMQ-Sfk8Y_45732_45809.png